Screen Share

When someone shares their screen in a CVI call, Raven can see that shared content the same way it sees the camera feed. The PAL can read on-screen text, notice errors, follow a walkthrough, and answer questions like “what tab am I on?” without you wiring up a custom vision tool. We recommend Raven-2 for screen sharing: it adds visual history within the call, dedicated screen-awareness queries, and screen tools. Limited screen perception is still available with Raven-1.

How it works

Short screen summaries are supplied to the conversational LLM alongside camera and audio context, so the PAL can respond to what the participant is sharing on its next turn.

Two different directionsScreen share here means the user shares with the PAL. That is different from Presentation, where the PAL publishes slides for participants to watch.

Requirements

Requirement	Detail
Raven on	Set `layers.perception.perception_model` to `raven-2`, or use `raven-1` (the default) for basic screen perception.
Participant shares	Someone must actually start screen share in the client.
Video conversation	Screen perception needs a live video room (not chat-only / audio-only).

With Raven on, no extra perception fields are required for basic “see my screen” behavior.

{
  "layers": {
    "perception": {
      "perception_model": "raven-2"
    }
  }
}

Screen sharing is available by default. To let participants use it, include a screen-share control in your frontend. The Conversation block includes one, or you can add ScreenShareButton from @tavus/cvi-ui to a custom layout.

const canShareScreen = user.role === "presenter";

return (
  <CallControls>
    {canShareScreen && <ScreenShareButton />}
  </CallControls>
);

To disallow screen sharing, leave the control out of your UI. You can use the same pattern to show it only for certain roles or workflows.

OptionalIf you do not need visual perception at all, set perception_model to "off". This also turns off camera perception.

What the PAL receives

With Raven-2, the latest screen and camera observations accompany the user’s words in a <user_visual_analysis> block. Visual history from the shared screen is supplied separately in system context. For a custom LLM, messages can include:

[
  {
    "role": "system",
    "content": "<screen_timeline>...</screen_timeline> <screen_events>...</screen_events>"
  },
  {
    "role": "user",
    "content": "<user_visual_analysis>Screen: A settings page is open.</user_visual_analysis>What page am I on?"
  }
]

Visual history from the camera can also appear as <camera_timeline> and <camera_events>. Sections appear when observations are available; their text is not a fixed JSON schema. With Raven-1, rolling screen descriptions instead appear in a system message inside <user_screen>. See custom LLM perception context. Raven-2 can use visual history to answer questions such as “What error was shown before I opened settings?” Visual history is scoped to the current conversation and may be incomplete or delayed. It does not guarantee that every frame, page, or value is remembered. Raven also answers prompted visual questions against the screen when the user is sharing (for example, “what error is on my screen?”), routing those to the screenshare stream instead of the webcam when that is the right source.

Make the PAL act on what it sees

Objectives with visual modality

For structured goals that depend on the shared screen, use objectives with "modality": "visual":

{
  "objective_name": "capture_error_code",
  "objective_prompt": "Check if an error message is visible on the user's shared screen. Extract the error code and message text.",
  "modality": "visual",
  "confirmation_mode": "auto"
}

Screen-awareness queries (Raven-2)

screen_awareness_queries focus Raven-2 on shared-screen content such as error dialogs, forms, or IDE state. Use visual_awareness_queries separately for the camera feed. Pair these queries with a system prompt that tells the PAL how to react when those cues appear.

{
  "layers": {
    "perception": {
      "perception_model": "raven-2",
      "screen_awareness_queries": [
        "Is an error dialog or stack trace visible on the shared screen?",
        "Is the user looking at a login or settings form?"
      ]
    }
  }
}

Screen tools (Raven-2)

Create a registry tool with origin: "screen" and put its trigger conditions in the tool’s description. Attach it to a PAL configured with raven-2. Screen tools run in parallel with the conversational LLM and emit conversation.perception_tool_call with modality: "screen". See Tool Calling for Perception for a complete example.

Frontend tips

Prefer the CVI UI Conversation block if you want share controls and sensible layout out of the box.
For a custom Daily React layout, show the shared screen as the main surface and keep face video available (often as PiP) while they share.
Ask users to share a specific window or tab when possible. Full-desktop shares work, but a focused tab is usually cleaner for the PAL to read.

Perception: Raven configuration
Models → Raven: model overview
Presentation: PAL shares slides to the user

Getting Started

Guides

Build

Deploy

Debug

Teams & SSO

Resources

Screen Share

How it works

Requirements

What the PAL receives

Make the PAL act on what it sees

Objectives with visual modality

Screen-awareness queries (Raven-2)

Screen tools (Raven-2)

Frontend tips

​How it works

​Requirements

​Control access to screen share

​What the PAL receives

​Make the PAL act on what it sees

​Objectives with visual modality

​Screen-awareness queries (Raven-2)

​Screen tools (Raven-2)

​Frontend tips

​Related

How it works

Requirements

Control access to screen share

What the PAL receives

Make the PAL act on what it sees

Objectives with visual modality

Screen-awareness queries (Raven-2)

Screen tools (Raven-2)

Frontend tips

Related