Perception

The Perception Layer gives your PAL real-time understanding of what participants show and how they sound. Raven supplies this context to the conversational LLM and can trigger tools from camera, shared-screen, or audio cues. Choose configuration based on when you need the result:

During the call: Use awareness queries to focus the PAL’s live observations and registry tools to act on detected cues.
After the call: Use perception_analysis_queries to shape the report delivered to your conversation callback.

Configuring the Perception Layer

Set these options in the PAL’s layers.perception object when you create or patch it.

`perception_model`

raven-2: Real-time emotional understanding from user audio, with improved camera and screen understanding for more context-aware responses. Raven-2 also introduces visual history within a conversation, dedicated screen-awareness queries and tools, and optional proactive callouts.
raven-1 (default): Real-time audio and visual perception, with limited screen understanding.
off: Disables the perception layer.

Raven-2

Raven-2 helps your PAL follow what is happening now and refer back to what was shown earlier in the same call:

Live understanding: Camera and screen observations distinguish the latest visible state from recent changes. Audio perception supplies tone and emotional cues when enabled.
Visual history: Camera and screen timelines let the PAL answer questions such as “What was the previous error?” or “Which object did I hold up first?”
Dedicated screen perception: screen_awareness_queries focus on shared-screen content, while tools with origin: "screen" act on screen-specific cues.
Callouts: With callouts: "on", the PAL can speak up about camera observations while the user is quiet. See Callouts.

Select Raven-2 explicitly; omitting perception_model still selects Raven-1:

{
  "layers": {
    "perception": {
      "perception_model": "raven-2"
    }
  }
}

Visual history is built automatically from observations during the conversation. To upgrade an existing PAL, patch the PAL to set layers.perception.perception_model to "raven-2", then start a new conversation.

During-call perception

Raven provides live perception without custom queries. Add awareness queries when the PAL should pay particular attention to something during the call. Their answers become context for the conversational LLM; they do not instruct it how to respond. Put response instructions in the PAL’s system_prompt. Live analysis is available through the Utterance event as user_visual_analysis and user_audio_analysis. See custom LLM perception context for how observations and visual history reach your model.

Visual Perception Configuration

`visual_awareness_queries`

An array of questions evaluated against the camera feed during the call. For Raven-2 shared-screen content, use screen_awareness_queries instead.

{
  "layers": {
    "perception": {
      "perception_model": "raven-2",
      "visual_awareness_queries": [
        "What object is the user holding up to the camera?",
        "Is more than one person visible?",
        "Is the user presenting a slide deck?"
      ]
    }
  }
}

Audio Perception Configuration

Raven-1 and Raven-2 analyze user tone and delivery. When emotion recognition is enabled, emotional cues also accompany utterances so the PAL can respond naturally. For example:

<user_audio_analysis>The user sounded sarcastic when they said this</user_audio_analysis>
Wow, I love Mondays.

Audio analysis tags are stripped from transcription callbacks.

`audio_awareness_queries`

An array of questions evaluated against the audio stream during the call. There is no separate post-call audio analysis report.

{
  "layers": {
    "perception": {
      "perception_model": "raven-2",
      "audio_awareness_queries": [
        "Is the user speaking quickly or pausing frequently?",
        "Is the user raising their voice?"
      ]
    }
  }
}

`emotion_recognition`

Controls whether Raven may infer emotion from biometric signals such as facial expression and tone of voice. See EU AI Act configuration.

full: Allows biometric emotion analysis.
limited: Disables biometric-derived emotion analysis. Other cues, such as spoken content, objects, and activity, still flow through.
auto (default): Follows the conversation’s policy: limited when policy is eu, otherwise full.

{
  "layers": {
    "perception": {
      "perception_model": "raven-2",
      "emotion_recognition": "auto"
    }
  }
}

Screen Perception Configuration (Raven-2)

We recommend Raven-2 for screen sharing. Limited screen perception remains available with Raven-1. A participant must start sharing their screen for Raven to observe it; see Screen Share.

`screen_awareness_queries`

An array of questions evaluated against the shared screen during the call. Requires perception_model: "raven-2".

{
  "layers": {
    "perception": {
      "perception_model": "raven-2",
      "screen_awareness_queries": [
        "Which app or page is currently visible?",
        "What error code and message are displayed?"
      ]
    }
  }
}

Perception tools

To trigger an action from a detected cue, create a registry tool and attach it to the PAL. Put the trigger conditions and extraction instructions in the tool’s description:

origin: "vision": Camera cues.
origin: "audio": Audio cues.
origin: "screen": Shared-screen cues; requires Raven-2.

Raven receives the attached tool definitions, including their descriptions and parameter schemas. You do not need a separate tool-prompt field to repeat those instructions. For example, a screen tool’s description can say: “Trigger when an error dialog is visible on the shared screen. Extract the error text exactly as displayed.” See Tool Calling for Perception for create-and-attach examples and delivery behavior.

Callouts (Raven-2)

Set callouts to "on" to let the PAL proactively speak up about camera observations while the user is quiet. The default is "off". Enabling callouts requires perception_model: "raven-2". Describe what to notice and what to say in the PAL’s top-level system_prompt, alongside layers when creating the PAL. visual_awareness_queries can focus its camera observations. For example:

{
  "pal_name": "Assembly Coach",
  "default_face_id": "rf4e9d9790f0",
  "system_prompt": "Help the user assemble the device. While they are quiet, speak up if they hold a part upside down in view of the camera. Identify the part and explain how to orient it. Stay quiet about unrelated background activity.",
  "layers": {
    "perception": {
      "perception_model": "raven-2",
      "callouts": "on",
      "visual_awareness_queries": [
        "Which part is the user holding, and how is it oriented?"
      ]
    }
  }
}

Post-call perception analysis

Use perception_analysis_queries for a report generated after the conversation ends. These queries shape the report, rather than the PAL’s live behavior. The report combines visual observations, including shared-screen observations when available, with the transcript and its audio analysis.

`perception_analysis_queries`

An array of questions for the post-call report. For example, a support PAL can request a summary of the errors and pages shown during screen sharing:

{
  "layers": {
    "perception": {
      "perception_model": "raven-2",
      "perception_analysis_queries": [
        "Which error codes and messages appeared on the shared screen?",
        "Which troubleshooting pages did the user show, and in what order?",
        "Was more than one person visible on camera during the call?"
      ]
    }
  }
}

Awareness queries are optional. You can request post-call analysis without setting visual_awareness_queries or screen_awareness_queries. Post-call queries are not automatically converted into live awareness queries; add live queries only when you also want to focus the PAL’s observations during the call.

Answers arrive in an application.perception_analysis event sent to your conversation callback. For example:

{
  "properties": {
    "analysis": "The shared screen showed error E104 on the connection page, followed by the network settings page. One participant was visible on camera."
  },
  "conversation_id": "<conversation_id>",
  "event_type": "application.perception_analysis",
  "timestamp": "2026-10-19T09:13:35.361736Z"
}

Use focused questions that support your PAL’s purpose. The report can only address details captured in the observations; it does not guarantee a complete record of every frame or page.

Legacy configuration

Inline visual_tools and audio_tools are deprecated for new integrations. Create tools in the registry and attach them to the PAL instead. See Legacy inline tool calling for existing integrations and migration instructions.

Existing integrations may still use separate perception tool-prompt fields. New integrations should put trigger conditions in the registry tool’s description; see legacy perception prompts.

Each awareness or post-call query and each legacy tool-prompt string has a 10,000-character limit. Longer entries are rejected by the API.

Getting Started

Guides

Build

Deploy

Debug

Teams & SSO

Resources

Configuring the Perception Layer

`perception_model`

Raven-2

During-call perception

Visual Perception Configuration

`visual_awareness_queries`

Audio Perception Configuration

`audio_awareness_queries`

`emotion_recognition`

Screen Perception Configuration (Raven-2)

`screen_awareness_queries`

Perception tools

Callouts (Raven-2)

Post-call perception analysis

`perception_analysis_queries`

Legacy configuration

​Configuring the Perception Layer

​perception_model

​Raven-2

​During-call perception

​Visual Perception Configuration

​visual_awareness_queries

​Audio Perception Configuration

​audio_awareness_queries

​emotion_recognition

​Screen Perception Configuration (Raven-2)

​screen_awareness_queries

​Perception tools

​Callouts (Raven-2)

​Post-call perception analysis

​perception_analysis_queries

​Legacy configuration

Configuring the Perception Layer

`perception_model`

Raven-2

During-call perception

Visual Perception Configuration

`visual_awareness_queries`

Audio Perception Configuration

`audio_awareness_queries`

`emotion_recognition`

Screen Perception Configuration (Raven-2)

`screen_awareness_queries`

Perception tools

Callouts (Raven-2)

Post-call perception analysis

`perception_analysis_queries`

Legacy configuration