Cluster: standby and load balancing
Two or more central computers can work as one KRZN Live:
- Standby: a second computer takes over when the main one stops (power cut, crash, frozen, network gone): VPS tunnel, site pages, intercom, recordings and NDI® outputs move to it in about 8 seconds, by themselves.
- Load balancing: the heavy work of the stream paths (NDI outputs, NDI and camera inputs, recordings) is spread over the computers, so that more sites can be received. A computer that stops hands its paths to the others.
There is always one main computer: it runs the tunnel and owns the settings. The other computers are its standby computers (standby mode) or its load-balancing computers. When the main computer stops, the computer with the highest priority becomes the main computer.
How it works
The computers never talk to each other directly: they all talk to the VPS edge, that every central computer already reaches. The edge keeps a lease: the computer that holds it is the main computer, and renews it every 2 seconds. When the main computer stops, its lease ends after 8 seconds and the next computer gets it.
VPS edge (hub: lease, configuration)
/ | \
PC-A (main) PC-B (standby) PC-C (standby)
tunnel, NDI ready to take over
- Fencing: a main computer that can't reach the edge for 8 seconds stops its tunnel and the NDI outputs of the edge feeds by itself, before another computer can take over: two computers are never main at the same time.
- Tunnel: every computer of the cluster connects to the frp server with the same session ID. When a new main computer connects, the frp server drops the session of the computer that stopped at once, instead of waiting for it to time out (30 to 40 seconds).
- NDI names: by default (
clusterSharedNDINames), the NDI name of a path doesn't contain the name of the computer:KRZN Live-NORTH (KRZN Live - North)instead ofPC-A (KRZN Live - North). Whichever computer publishes it, vMix finds the same source: its inputs keep working after a takeover. Each path has its own machine name, so two computers never announce the same name. - Configuration: the main computer sends its configuration to the edge, encrypted with the key of the cluster (the edge can't read it), with the site pages, the presets, the slates and the media files. The other computers apply it within a few seconds, keeping their own settings (see below).
Requirements
- The VPS edge, version 1.5 or later, with
clusterHub: yesin itsedge.yml(the installer adds it). - Sites must send through the VPS edge. Sites that send to the home address of central (Home route) can't fail over: the router sends them to one computer. Stream Paths shows "home route: not covered by failover" on them.
- Cameras and capture cards plugged into a computer stay on it.
- Each computer needs the NDI runtime and FFmpeg (
Install NDI and FFmpeg.bat), and should be on the same network as vMix. - The computers read each other's streams with RTSP (port 8554) on the local network: allow it in the Windows firewall.
Create a cluster
On the computer that will be the main one: Settings > Cluster, choose Standby or Load balancing, then Create a cluster. The cluster is created on the edge.
Add a computer
- On the main computer: Status > Cluster > Add a PC. A join code is shown, as text and as a QR code. It contains the key of the cluster and the password of the edge: share it only with the computers of your setup.
- On the new computer: install KRZN Live (download page), open Settings > Cluster, paste the code in Join a cluster.
- The new computer receives the settings and stream paths of the main computer within seconds (a copy of its previous configuration file is kept as
krznlive.yml.bak-before-cluster-...) and appears in the cluster panel.
A computer installed from the download page always starts alone: it never contains the key of the cluster.
Standby mode
| Main computer | Standby computers | |
|---|---|---|
| Tunnel, site pages, intercom | yes | no |
| Edge feeds | pulled, published as NDI, recorded | hot: pulled and recorded (dual recording), not published as NDI. warm: not pulled |
| Other paths (home route, inputs) | run | read from the main computer when watched |
- Hot standby (default): the takeover only starts the NDI outputs, the feeds are already received. The VPS sends each feed a second time: at 6 Mbit/s per site, 6 sites cost about 16 GB per hour of service (about USD 1.50 per hour on AWS). The settings show the estimate. Only the first standby computer needs to be hot.
- Warm standby: no extra traffic; the feeds are pulled at the takeover (2 to 3 seconds more).
- Dual recording (hot standby): the standby computer records the same feeds. The Recordings page shows "also on PC-B": if the main computer stops in the middle of a recording, the copy of the standby computer is complete.
- Hand back: when the preferred computer (highest priority) comes back,
automakes it the main computer again once it has been healthy for 5 minutes and no feed is live or recorded;manualwaits for Make main.
Load balancing mode
Each path runs on one computer (its owner): it pulls the feed, publishes the NDI output, receives the NDI or camera input, records. The other computers read the path from the owner only when someone watches it (multiviewer, thumbnails).
- New paths go to the computer with the most free capacity. A short speed test runs once on each computer to know the cost of an NDI output there.
- A path keeps its computer. Paths move only when their computer stops (at once, even live), with Rebalance (paths that aren't live), or, with
clusterAutoBalance, when a computer is overloaded and nothing is live. - Runs on in Stream Paths pins a path to a computer. Paths with a camera or capture card are pinned to their computer automatically: add them from the dashboard of that computer, they're sent to the main computer.
- Spare NDI output publishes the NDI output of an important path on a second computer too, under the name of that computer: vMix has both sources, for an instant switch.
- Only coordinate keeps the main computer free of paths (for a weak main computer, or one that also runs vMix).
- Max NDI outputs limits the NDI outputs of a computer.
The cluster panel
Status > Cluster shows every computer: role, heartbeat, version, CPU, GPU, memory, disk, what it runs. Buttons:
- Make main: planned switch (about 2 to 5 seconds: the tunnel is released cleanly first).
- Test failover: makes the standby computer the main computer for real, to practise, and shows the measured switch time.
- Rebalance (load balancing).
- Update the cluster: installs the downloaded update on the other computers one after the other, then hands over and updates the main computer. The cluster keeps working.
- Remove: removes a computer; the key of the cluster changes, and the removed computer can't come back without a new join code.
On a computer that isn't the main one, the shared settings and the stream paths are read-only (a banner says where to change them).
Settings
| Setting | Default | Shared | Meaning |
|---|---|---|---|
cluster |
none |
yes | none, standby or balance |
clusterName |
krznlive-main |
no | name of the cluster: several clusters can use the same edge |
clusterNodeName |
empty | no | name of this computer (empty: the name of the computer) |
clusterKey |
empty | no | shared secret, given by the join code |
clusterPriority |
10 | no | higher: preferred main computer |
clusterCanBeMain |
true | no | false: this computer only works for the others |
clusterHandBack |
auto |
yes | auto or manual |
clusterStandby |
hot |
yes | hot or warm |
clusterDualRecord |
true | yes | hot standby computers record too |
clusterAutoBalance |
false | yes | move paths of an overloaded computer, when nothing is live |
clusterCoordinateOnly |
false | no | the main computer runs no path (load balancing) |
clusterMaxNDIOutputs |
0 | no | at most this many NDI outputs here (0: no limit) |
clusterSharedNDINames |
true | yes | NDI names without the computer name |
clusterLANAddress |
empty | no | address used by the other computers to read the streams of this one (empty: automatic) |
clusterHub |
false | no | on the edge only |
path clusterPin |
empty | yes | the path always runs on this computer (ID or name) |
path clusterSpare |
false | yes | spare NDI output on a second computer |
Settings that belong to each computer are never copied: the ones above marked "no", the NDI runtime and FFmpeg folders, the video decoding, the dashboard address, the window and tray options, logs and updates.
Alerts
Only the main computer sends alerts, so they don't arrive once per computer. A takeover is reported with its measured times, for example: PC-A stopped answering at 10:42:15; PC-B took over (NDI after 10.9 s, tunnel after 8.2 s) (counted from the last heartbeat of PC-A).
Measured (two computers, on one test machine)
| Case | Result |
|---|---|
| Main computer killed (power cut) | new main computer after 7 to 8 s; tunnel at once (0.1 s after); NDI picture 2.8 s after (hot standby) |
| Planned switch (Make main, Test failover) | 1 to 2 s for the lease and the tunnel, 4 to 5 s until the NDI picture |
| Main computer loses the edge | it stops its tunnel and NDI outputs after 7.7 s, before the lease ends (8 s) |
| Edge restarted | nobody takes over: the main computer keeps its lease |
| Edge down for everyone | every computer stops what could conflict; when it's back, the computer with the highest priority is main again |
| Load balancing, a computer stops | its paths move after about 6 s, live with NDI after about 7.5 s |
Limits
- Home-route sites and cameras plugged into a computer can't move to another computer.
- Viewers of the site pages, the intercom and the remote dashboard reconnect after a takeover.
- The cluster is coordinated by the VPS: when it's down, nobody takes over (the feeds go through it anyway).