Real-time transcription streams transcript data while the meeting is in progress. Enable it by selecting a real-time transcription model. ## Key Features - Live transcript streaming via WebSocket connection - Immediate access to spoken content as it happens - Suitable for applications requiring real-time processing - Optional real-time audio streaming for raw audio access ## How it works After creating a meeting bot with a **real-time transcription model**, or enabling the **Realtime features** add-on for another model, the response includes `websocket_url` and `websocket_read_only_url` for event streaming. - **`websocket_url`**: Full access, allows [Actions](#websocket-actions). - **`websocket_read_only_url`**: Read-only access. This is safe to provide to your customer's front-end directly. Connect to either URL to receive live updates. The first `connected` event contains cached transcript and chat history, the latest state for every observed participant, and the current bot status. Subsequent events keep that context current. Only `websocket_url` accepts actions. Realtime features also include `websocket_audio_url` for a separate raw-audio connection. ## Manual Recording Start Set `recording_start_mode: "manual"` when the bot should join without immediately recording. Manual mode requires either a real-time transcription model or the Realtime features add-on (`realtime_audio: true`); requests without either are rejected. After connecting to `websocket_url`, use the `connected` snapshot and subsequent events to apply your own participant and consent policy. When your conditions are satisfied, send the [`start-recording`](#start-recording) action. Skribby provides the context and action but does not decide whether consent was given. ## Realtime Audio Streaming You can also receive raw audio data in real time through a separate WebSocket connection. ### Enabling Realtime Features for Raw Audio Real-time models include Realtime features by default. To enable the same event, action, and raw-audio bundle with another model, set the backwards-compatible `realtime_audio: true` field in your request: ```json { "transcription_model": "none", "meeting_url": "https://meet.google.com/abc-defg-hij", "service": "gmeet", "bot_name": "Alex from Acme", "realtime_audio": true } ``` The bot response will include a `websocket_audio_url` field containing the WebSocket URL for receiving audio data. ### Using the SDK ```ts // Create bot with Realtime features enabled const bot = await client.createBot({ transcription_model: 'none', meeting_url: 'https://meet.google.com/abc-defg-hij', service: 'gmeet', bot_name: 'Alex from Acme', realtime_audio: true, }); // Get the realtime client (includes audio by default) const realtimeClient = bot.getRealtimeClient(); // Listen to audio events realtimeClient.on('audio', (buffer: Buffer) => { // buffer is 16-bit PCM audio at 16kHz sample rate processAudio(buffer); }); await realtimeClient.connect(); // Check audio connection status console.log('Audio connected:', realtimeClient.audioConnected); ``` ### Without Audio Streaming If you don't need audio streaming, you can get a realtime client without it: ```ts // Get realtime client WITHOUT audio streaming const transcriptOnlyClient = bot.getRealtimeClient(true); await transcriptOnlyClient.connect(); ``` ### Audio Format The audio data received via the `audio` event is: - **Format:** 16-bit PCM (signed, little-endian) - **Sample Rate:** 16kHz - **Channels:** Mono ## WebSocket Events Events arrive as JSON-encoded WebSocket messages with this structure: ```json { "type": "[event]", "data": {...} } ``` ### Connected When you connect, including through `websocket_read_only_url`, the first `connected` event contains cached transcript and chat history, the latest state for every observed participant, and the current bot status. Reconnecting reads this server-side snapshot and does not require the bot to resend its history. ```json { "type": "connected", "data": { "transcripts": [ { "transcript": "This contains the spoken text.", "start": 1.23, "end": 4.56, "speaker": 0, "speaker_name": "John Doe" } ], "participants": [ { "participantId": "participant-123", "participantName": "Ada Lovelace", "timestamp": 1784550725000, "lastSeenAt": 1784550725000, "state": { "active": true, "microphone": "muted", "camera": "off", "screenshare": "not-sharing" } } ], "chat_messages": [ { "id": "message-123", "parent_id": null, "username": "Ada Lovelace", "content": "I consent to recording.", "user_avatar": null } ], "status": "waiting_to_record" } } ``` #### SDK tip: full transcript buffer If you're using the TypeScript/JavaScript SDK, the `RealtimeClient` maintains an internal transcript buffer for you: - On `connect()`, the SDK resets the buffer. - When the `"connected"` snapshot arrives, the SDK seeds the buffer from `data.transcripts`. - On each `"ts"` event, the SDK appends the new segment. You can read the full transcript so far at any time via `realtimeClient.transcript`: ```ts const realtimeClient = bot.getRealtimeClient(); realtimeClient.on('ts', (segment) => { console.log(segment.speaker_name, segment.transcript); console.log('Transcript so far:', realtimeClient.transcript); }); await realtimeClient.connect(); ``` ### Start The `start` event indicates that the bot has joined and started recording. Transcript events can follow from this point. See [Bot Lifecycle](./bot-lifecycle.md) for the joining process. ```json { "type": "start" } ``` ### Status Update The `status-update` event reports the previous and current bot status. If `new_status` becomes `finished` or `not_admitted`, a third field called `stop_reason` will also be included. Refer to the [Bot Lifecycle Documentation](./bot-lifecycle.md#stop-reasons) for all stop reason codes. ```json { "type": "status-update", "data": { "old_status": "joining", "new_status": "recording" } } ``` When the bot stops, `stop_reason` is included: ```json { "type": "status-update", "data": { "old_status": "processing", "new_status": "finished", "stop_reason": "meeting_ended" } } ``` ### Recording Started The `recording-started` event confirms that media capture began. For a manual-start bot, it acknowledges a successful `start-recording` action. `started_at` is an epoch-millisecond timestamp. A `status-update` to `recording` also confirms the transition. ```json { "type": "recording-started", "data": { "started_at": 1784550726123 } } ``` ### Transcript The `ts` event contains a live transcript segment with timestamps and speaker information. Early segments may use a generic `speaker_name`, such as `"Speaker 1"`, while Skribby correlates the audio stream with the platform participant list. Later segments use the participant name when that match is available. ```json { "type": "ts", "data": { "transcript": "This contains the spoken text.", "start": 1.23, "end": 4.56, "speaker": 0, "speaker_name": "John Doe" } } ``` ### Chat Message The `chat-message` event is emitted when a new chat message is received. **Zoom only:** chat message events include `id` and `parent_id` for thread support, and may include a `to` field for the recipient name. Refer to the [Zoom Chat Guide](./zoom-chat.md) for details on how these fields work across Meetings and Webinars. ```json { "type": "chat-message", "data": { "id": "msg_8f1c8f0", // Zoom only "parent_id": "msg_7c2b4a9", // Zoom only "username": "John Doe", "to": "Jane Doe", // Zoom only "content": "Foo bar.", "user_avatar": null // Either a URL or null. Do not rely on this field for permanent access. } } ``` **Field notes:** - `id`: unique message identifier. - `parent_id`: (Optional, Zoom only) parent message ID for threaded replies, otherwise `null`. - `username`: name of the sender. - `to`: (Optional, Zoom only) populated when Zoom exposes a recipient. For private messages, this is usually the recipient name. For public messages, it may be `"Meeting Group Chat"`, `"Everyone"`, or absent. ### Participant Events Skribby emits realtime events when participants join, leave, speak, or change their meeting state: - **`participant-tracked`**: A participant was detected for the first time. This also fires for participants already present when the bot joins. - **`started-speaking`**: A participant started speaking. - **`stopped-speaking`**: A participant stopped speaking. - **`participant-left`**: A participant left the meeting. - **`participant-rejoined`**: A participant rejoined the meeting. - **`participant-muted`**: A participant muted their microphone. - **`participant-unmuted`**: A participant unmuted their microphone. - **`participant-camera-on`**: A participant turned their camera on. - **`participant-camera-off`**: A participant turned their camera off. - **`participant-started-screenshare`**: A participant started sharing their screen. - **`participant-stopped-screenshare`**: A participant stopped sharing their screen. Example: ```json { "type": "participant-muted", "data": { "participantId": "John Doe", "participantName": "John Doe", "timestamp": 1750820602963, "state": { "active": true, "microphone": "muted", "camera": "off", "screenshare": "unknown" }, "lastSeenAt": 1750820602963 } } ``` The same event is added to the participant's stored timeline with the same timestamp. For example, `participant-muted` is stored as `{"type":"muted","timestamp":1750820602963}`. See [Participant Timelines](./speaker-timelines.md) for retrieval details. - `state` is the current state after the event: `active` is a boolean; `microphone` is `muted`, `unmuted`, or `unknown`; `camera` is `on`, `off`, or `unknown`; and `screenshare` is `sharing`, `not-sharing`, or `unknown`. - `lastSeenAt` is the epoch-millisecond time when the participant was last seen in the meeting. ### Stop The `stop` event indicates that recording has ended. No further transcript events will be sent, so the client can disconnect. ```json { "type": "stop" } ``` ### Error The `error` event indicates that transcription has failed. No further transcript events will be sent. ```json { "type": "error", "data": { "message": "Error message" } } ``` ## WebSocket Actions You can also interact with the bot through WebSocket actions. Actions use the JSON structure below. WebSocket messages are strings, so serialize the JSON before sending it. Actions with missing required data are ignored. ```json { "action": "[action]", "data": {...} } ``` > **Important to remember** > Actions are only available through `websocket_url`. Messages sent through `websocket_read_only_url` are not executed. ### Start Recording For a bot created with `recording_start_mode: "manual"`, send this action after your application has decided its participant and consent requirements are met: ```json { "action": "start-recording" } ``` Listen for the `recording-started` event or a `status-update` to `recording` before treating recording as active. With the SDK, call `realtimeClient.startRecording()`. Repeated actions are safe: recording starts only once. ### Send Chat Message Use this action to send a message to the meeting chat. **Zoom only:** include `to` to send a message to a specific recipient, or `reply_to_message` to reply inside a thread. Refer to the [Zoom Chat Guide](./zoom-chat.md) for important behavior notes regarding Meetings vs Webinars. ```json { "action": "chat-message", "data": { "content": "Welcome to the meeting!", "to": "Meeting Group Chat", // Zoom only "reply_to_message": "msg_7c2b4a9" // Zoom only } } ``` **Important behavior notes:** - `to` is **Zoom-only**. To message everyone, use `"Meeting Group Chat"` for meetings and `"Everyone"` for webinars. - If the named Zoom recipient is not found, the bot will **skip sending** the message. - `to` is intended for **non-threaded** sends. Do not combine `to` and `reply_to_message`. ### Stop the bot This action stops the bot. ```json { "action": "stop" } ```