Connection log
Every computer running KRZN Live keeps a journal of its connection problems, and the VPS shows the journals of all computers on one page, together with what the VPS itself saw. The page answers one question: when the connection dropped, was it the site (its Internet, its computer, KRZN Live), the tunnel, or the VPS?
Open it from anywhere: https://call.artsministy.online/netlog/ (password protected).
What a computer records
The journal is written to the netlog folder next to the configuration, one file per day (YYYY-MM-DD.jsonl), and kept for 90 days. It contains:
- every warning and error of the log, and every log entry about a connection (closed, reconnect, timeout, tunnel, frp…);
- the result of a probe that runs every 5 seconds and checks the VPS (the frp server, or the Go Live server) and two neutral hosts (Cloudflare 1.1.1.1 and Google 8.8.8.8). Each outage gets a cause:
| Cause | Meaning |
|---|---|
| Router / Wi-Fi / cable | the computer has no network, or its router doesn't answer pings any more: Wi-Fi, cable, or the router itself (off, restarting) |
| Internet of the site down | the router answers, but the VPS, Cloudflare and Google don't: the Internet provider |
| Network changed | the computer moved to another network: cable to Wi-Fi, another Wi-Fi, a phone hotspot (detected by its name, its address or its USB driver) |
| DNS (names) failed | the name of the VPS can't be found while the Internet works: the DNS of the router or of the provider |
| Internet provider switched | the public address of the site changed (failover between providers, mobile data, router reconnect) |
| Computer asleep / frozen | the computer stopped running for 45 s or more |
| KRZN Live restarted / crashed | KRZN Live stopped, or did not stop normally the last time; the lines of krznlive-crash.log are imported |
| Tunnel only | the tunnel is down while the VPS and the Internet answer |
| VPS / route to the VPS | the VPS doesn't answer while Cloudflare and Google do |
| Slow link to the VPS | connecting to the VPS takes more than 800 ms (saturated upload, congested route) |
| Weak / unstable Wi-Fi | signal under 40%, roaming between access points, pings of the router lost or slow (not counted as drops) |
To tell where a connection broke, the computer also:
- pings its router every 5 seconds (a good link stays under 10 ms);
- reads its network every 30 seconds: adapter, router, Wi-Fi name, access point, signal, channel, speed;
- measures the route to the VPS (tracert) when the Internet, the VPS or the tunnel drops (at most once every 10 minutes), and the normal route a minute after start and every 12 hours, to compare. The VPS doesn't answer traceroute itself: a normal route ends with hops that don't answer;
- reports every 5 minutes when the link to the VPS or to the router loses answers.
The page shows the network of each computer (cable, Wi-Fi and signal, phone hotspot, ping of the router, time to the VPS), and a computer that stops reporting for 2 minutes.
The journal is sent to the VPS every 15 seconds. When the VPS can't be reached, it waits on the computer and is sent later: the page also shows the outages that prevented sending it at the time.
Settings:
# keep the journal and send it to the VPS (Windows computers only)
netlog: true
# the collector of the VPS (the key is the Go Live password)
netlogURL: https://call.artsministy.online/netlog/
What the VPS records
The collector (scripts/netlog, service krznlive-netlog) also watches the VPS:
- the sessions of the frp server: which computer connected, from which address, how long the session lasted, ports refused because an old session still held them, stalled connections;
- the limits of AWS: packets dropped because the instance went over its bandwidth, packets per second or connection allowance (ENA counters);
- CPU steal (the t3 CPU credits are exhausted), CPU saturation, memory;
- the services (frps, Go Live, edge, nginx): stopped, restarted, crashed; and reboots of the VPS.
Refused logins of strangers on the frp port (Internet scanners with a wrong token) are counted per hour.
Install or update it on the VPS:
sudo NETLOG_INGEST_KEY=<goLivePass> sh scripts/netlog/install-netlog.sh
The password of the page is generated at the first install and kept in /etc/krznlive-netlog.env.
Tunnel sessions
A computer alone (not in a cluster) now uses a session ID of its own on the frp server, that stays the same when KRZN Live or the tunnel restarts. The frp server then gives the ports back at once, instead of refusing them ("proxy already exists") until the old session times out.