Skribby
GuidesUpdated 1 week ago
Open .md

Realtime Transcription

Real-time transcription streams transcript data while the meeting is in progress. Enable it by selecting a real-time transcription model.

Key Features

  • Live transcript streaming via WebSocket connection
  • Immediate access to spoken content as it happens
  • Suitable for applications requiring real-time processing
  • Optional real-time audio streaming for raw audio access

How it works

After creating a meeting bot with a real-time transcription model, or enabling the Realtime features add-on for another model, the response includes websocket_url and websocket_read_only_url for event streaming.

  • websocket_url: Full access, allows Actions.
  • websocket_read_only_url: Read-only access. This is safe to provide to your customer's front-end directly.

Connect to either URL to receive live updates. The first connected event contains cached transcript and chat history, the latest state for every observed participant, and the current bot status. Subsequent events keep that context current. Only websocket_url accepts actions.

Realtime features also include websocket_audio_url for a separate raw-audio connection.

Manual Recording Start

Set recording_start_mode: "manual" when the bot should join without immediately recording. Manual mode requires either a real-time transcription model or the Realtime features add-on (realtime_audio: true); requests without either are rejected.

After connecting to websocket_url, use the connected snapshot and subsequent events to apply your own participant and consent policy. When your conditions are satisfied, send the start-recording action. Skribby provides the context and action but does not decide whether consent was given.

Realtime Audio Streaming

You can also receive raw audio data in real time through a separate WebSocket connection.

Enabling Realtime Features for Raw Audio

Real-time models include Realtime features by default. To enable the same event, action, and raw-audio bundle with another model, set the backwards-compatible realtime_audio: true field in your request:

{
    "transcription_model": "none",
    "meeting_url": "https://meet.google.com/abc-defg-hij",
    "service": "gmeet",
    "bot_name": "Alex from Acme",
    "realtime_audio": true
}

The bot response will include a websocket_audio_url field containing the WebSocket URL for receiving audio data.

Using the SDK

// Create bot with Realtime features enabled
const bot = await client.createBot({
    transcription_model: 'none',
    meeting_url: 'https://meet.google.com/abc-defg-hij',
    service: 'gmeet',
    bot_name: 'Alex from Acme',
    realtime_audio: true,
});

// Get the realtime client (includes audio by default)
const realtimeClient = bot.getRealtimeClient();

// Listen to audio events
realtimeClient.on('audio', (buffer: Buffer) => {
    // buffer is 16-bit PCM audio at 16kHz sample rate
    processAudio(buffer);
});

await realtimeClient.connect();

// Check audio connection status
console.log('Audio connected:', realtimeClient.audioConnected);

Without Audio Streaming

If you don't need audio streaming, you can get a realtime client without it:

// Get realtime client WITHOUT audio streaming
const transcriptOnlyClient = bot.getRealtimeClient(true);
await transcriptOnlyClient.connect();

Audio Format

The audio data received via the audio event is:

  • Format: 16-bit PCM (signed, little-endian)
  • Sample Rate: 16kHz
  • Channels: Mono

WebSocket Events

Events arrive as JSON-encoded WebSocket messages with this structure:

{
  "type": "[event]",
  "data": {...}
}

Connected

When you connect, including through websocket_read_only_url, the first connected event contains cached transcript and chat history, the latest state for every observed participant, and the current bot status. Reconnecting reads this server-side snapshot and does not require the bot to resend its history.

{
    "type": "connected",
    "data": {
        "transcripts": [
            {
                "transcript": "This contains the spoken text.",
                "start": 1.23,
                "end": 4.56,
                "speaker": 0,
                "speaker_name": "John Doe"
            }
        ],
        "participants": [
            {
                "participantId": "participant-123",
                "participantName": "Ada Lovelace",
                "timestamp": 1784550725000,
                "lastSeenAt": 1784550725000,
                "state": {
                    "active": true,
                    "microphone": "muted",
                    "camera": "off",
                    "screenshare": "not-sharing"
                }
            }
        ],
        "chat_messages": [
            {
                "id": "message-123",
                "parent_id": null,
                "username": "Ada Lovelace",
                "content": "I consent to recording.",
                "user_avatar": null
            }
        ],
        "status": "waiting_to_record"
    }
}

SDK tip: full transcript buffer

If you're using the TypeScript/JavaScript SDK, the RealtimeClient maintains an internal transcript buffer for you:

  • On connect(), the SDK resets the buffer.
  • When the "connected" snapshot arrives, the SDK seeds the buffer from data.transcripts.
  • On each "ts" event, the SDK appends the new segment.

You can read the full transcript so far at any time via realtimeClient.transcript:

const realtimeClient = bot.getRealtimeClient();
realtimeClient.on('ts', (segment) => {
    console.log(segment.speaker_name, segment.transcript);
    console.log('Transcript so far:', realtimeClient.transcript);
});
await realtimeClient.connect();

Start

The start event indicates that the bot has joined and started recording. Transcript events can follow from this point. See Bot Lifecycle for the joining process.

{
    "type": "start"
}

Status Update

The status-update event reports the previous and current bot status.

If new_status becomes finished or not_admitted, a third field called stop_reason will also be included. Refer to the Bot Lifecycle Documentation for all stop reason codes.

{
    "type": "status-update",
    "data": {
        "old_status": "joining",
        "new_status": "recording"
    }
}

When the bot stops, stop_reason is included:

{
    "type": "status-update",
    "data": {
        "old_status": "processing",
        "new_status": "finished",
        "stop_reason": "meeting_ended"
    }
}

Recording Started

The recording-started event confirms that media capture began. For a manual-start bot, it acknowledges a successful start-recording action. started_at is an epoch-millisecond timestamp. A status-update to recording also confirms the transition.

{
    "type": "recording-started",
    "data": {
        "started_at": 1784550726123
    }
}

Transcript

The ts event contains a live transcript segment with timestamps and speaker information. Early segments may use a generic speaker_name, such as "Speaker 1", while Skribby correlates the audio stream with the platform participant list. Later segments use the participant name when that match is available.

{
    "type": "ts",
    "data": {
        "transcript": "This contains the spoken text.",
        "start": 1.23,
        "end": 4.56,
        "speaker": 0,
        "speaker_name": "John Doe"
    }
}

Chat Message

The chat-message event is emitted when a new chat message is received.

Zoom only: chat message events include id and parent_id for thread support, and may include a to field for the recipient name. Refer to the Zoom Chat Guide for details on how these fields work across Meetings and Webinars.

{
    "type": "chat-message",
    "data": {
        "id": "msg_8f1c8f0", // Zoom only
        "parent_id": "msg_7c2b4a9", // Zoom only
        "username": "John Doe",
        "to": "Jane Doe", // Zoom only
        "content": "Foo bar.",
        "user_avatar": null // Either a URL or null. Do not rely on this field for permanent access.
    }
}

Field notes:

  • id: unique message identifier.
  • parent_id: (Optional, Zoom only) parent message ID for threaded replies, otherwise null.
  • username: name of the sender.
  • to: (Optional, Zoom only) populated when Zoom exposes a recipient. For private messages, this is usually the recipient name. For public messages, it may be "Meeting Group Chat", "Everyone", or absent.

Participant Events

Skribby emits realtime events when participants join, leave, speak, or change their meeting state:

  • participant-tracked: A participant was detected for the first time. This also fires for participants already present when the bot joins.
  • started-speaking: A participant started speaking.
  • stopped-speaking: A participant stopped speaking.
  • participant-left: A participant left the meeting.
  • participant-rejoined: A participant rejoined the meeting.
  • participant-muted: A participant muted their microphone.
  • participant-unmuted: A participant unmuted their microphone.
  • participant-camera-on: A participant turned their camera on.
  • participant-camera-off: A participant turned their camera off.
  • participant-started-screenshare: A participant started sharing their screen.
  • participant-stopped-screenshare: A participant stopped sharing their screen.

Example:

{
    "type": "participant-muted",
    "data": {
        "participantId": "John Doe",
        "participantName": "John Doe",
        "timestamp": 1750820602963,
        "state": {
            "active": true,
            "microphone": "muted",
            "camera": "off",
            "screenshare": "unknown"
        },
        "lastSeenAt": 1750820602963
    }
}

The same event is added to the participant's stored timeline with the same timestamp. For example, participant-muted is stored as {"type":"muted","timestamp":1750820602963}. See Participant Timelines for retrieval details.

  • state is the current state after the event: active is a boolean; microphone is muted, unmuted, or unknown; camera is on, off, or unknown; and screenshare is sharing, not-sharing, or unknown.
  • lastSeenAt is the epoch-millisecond time when the participant was last seen in the meeting.

Stop

The stop event indicates that recording has ended. No further transcript events will be sent, so the client can disconnect.

{
    "type": "stop"
}

Error

The error event indicates that transcription has failed. No further transcript events will be sent.

{
    "type": "error",
    "data": {
        "message": "Error message"
    }
}

WebSocket Actions

You can also interact with the bot through WebSocket actions. Actions use the JSON structure below. WebSocket messages are strings, so serialize the JSON before sending it. Actions with missing required data are ignored.

{
  "action": "[action]",
  "data": {...}
}

Start Recording

For a bot created with recording_start_mode: "manual", send this action after your application has decided its participant and consent requirements are met:

{
    "action": "start-recording"
}

Listen for the recording-started event or a status-update to recording before treating recording as active.

With the SDK, call realtimeClient.startRecording(). Repeated actions are safe: recording starts only once.

Send Chat Message

Use this action to send a message to the meeting chat.

Zoom only: include to to send a message to a specific recipient, or reply_to_message to reply inside a thread. Refer to the Zoom Chat Guide for important behavior notes regarding Meetings vs Webinars.

{
    "action": "chat-message",
    "data": {
        "content": "Welcome to the meeting!",
        "to": "Meeting Group Chat", // Zoom only
        "reply_to_message": "msg_7c2b4a9" // Zoom only
    }
}

Important behavior notes:

  • to is Zoom-only. To message everyone, use "Meeting Group Chat" for meetings and "Everyone" for webinars.
  • If the named Zoom recipient is not found, the bot will skip sending the message.
  • to is intended for non-threaded sends. Do not combine to and reply_to_message.

Stop the bot

This action stops the bot.

{
    "action": "stop"
}