Live captions
KRZN Live writes captions of the speech while it happens, with Whisper running on the central computer (no cloud service, no cost). The operator switches them on when the speech starts and off for the praise and music. The captions go into a vMix title and/or are drawn on the picture sent to Go Live, and each speech is saved as subtitles (.srt) and text (.txt).
How it works
Two models work together, so the words appear quickly and are still written well:
| Stage | Model | Runs on | Delay | Quality |
|---|---|---|---|---|
| Draft | Whisper small | CPU | about 1 s after the words (up to 3 s when the computer is busy) | fair: some words wrong |
| Correction | Whisper large-v3-turbo | graphics card (Vulkan) | the sentence is replaced about 6 s after the pause (7-12 s while the burn-in also uses the card) | very good, Taglish included |
While a sentence is spoken, the draft is written again every 0.4 s of new sound. At a pause, the sentence goes to the accurate model and its text replaces the draft. Without a graphics card, the drafts are not corrected.
Setting up
- Settings > Live captions: turn on Live captions, choose the path with the sound of the speech (empty: the Go Live program), add the words to know (names of people, places and products), and Apply.
- Captions page: click Download (the engine, 51 MB, and the models, 764 MB: once).
- Choose where the captions go:
- vMix title (recommended): add a title input in vMix (a lower third), turn on vMix Settings > Web Controller, and enter the title's name in Settings > Live captions. The title is put in the overlay channel while speech and taken out during the music. vMix's program then has the captions: nothing else to do.
- In the picture sent to Go Live (without vMix titles): KRZN Live draws the captions on the program and encodes it again (path
program-cc, graphics card encoder). Go Live sends it once it has worked for 10 s; if it stops, Go Live sends the plain program again by itself. Don't use both, or the captions show twice.
During the service
On the Captions page: press Speech when the speech starts, Music when the songs start. Music makes Whisper write nonsense: keep captions off during the songs. The page shows the captions as they are sent, the sound received, and the state of both models. ON is shown next to Captions in the menu while speech.
Files
| What | Where |
|---|---|
| Engine and models | captions\ next to krznlive.yml |
| Captions of each speech | recordings\captions\<date time>.srt and .txt |
Settings
| Setting | Default | Meaning |
|---|---|---|
captions |
false |
The whole feature. |
captionsSource |
"" |
Path whose sound is captioned (empty: goLiveSource). |
captionsModel / captionsDraftModel |
turbo / small | Accurate and fast models (empty: not used). Each computer of a cluster has its own. |
captionsGPU |
true |
Run the accurate model on the graphics card. |
captionsLanguage |
tl |
tl (Tagalog and Taglish), en, auto. |
captionsWords |
[] |
Words to know. |
captionsLines, captionsLineLength, captionsClearAfter |
2, 42, 4s | Lines shown, characters per line, cleared after this much silence. |
captionsVMix, captionsVMixAddress, captionsVMixInput, captionsVMixField, captionsVMixOverlay |
off, http://127.0.0.1:8088, "", first field, 1 |
vMix title. Overlay 0: KRZN Live doesn't show or hide the title. |
captionsBurnIn, captionsBurnInEncoder |
off, auto | Draw the captions on the program sent to Go Live. |
captionsTranscript |
true |
Save .srt and .txt. |