The LLM Layer in Tavus enables your persona to generate intelligent, context-aware responses. You can use Tavus-hosted models or connect your own OpenAI-compatible LLM.

​
Tavus-Hosted Models

​
1. model

Select one of the available models:
tavus-llama-4 is the default model and runs an optimized variant of Llama-4-17B.
  • tavus-llama-4 (Recommended)
  • tavus-gpt-4o
  • tavus-gpt-4o-mini
Context Window Limit
  • All Tavus-hosted models have a limit of 32,000 tokens.
  • Contexts over 25,000 tokens will experience noticeable performance degradation (slow response times).
Tip: 1 token ≈ 4 characters, therefore 32,000 tokens ≈ 128,000 characters (including spaces and punctuation).
"model": "tavus-llama-4"

​
2. tools

Optionally enable tool calling by defining functions the LLM can invoke.
Please see LLM Tool Calling for more details.

​
3. speculative_inference

When set to true, the LLM begins processing speech transcriptions before user input ends, improving responsiveness. 
"speculative_inference": true
This is field is optional, but recommended for better performance.

​
Example Configuration

{
  "persona_name": "Health Coach",
  "system_prompt": "You provide wellness tips and encouragement for people pursuing a healthy lifestyle.",
  "context": "You specialize in daily routines, diet advice, and motivational support.",
  "pipeline_mode": "full",
  "default_replica_id": "r665388ec672",
  "layers": {
    "llm": {
      "model": "tavus-gpt-4o",
      "speculative_inference": true
    }
  }
}

​
Custom LLMs

​
Prerequisites

To use your own OpenAI-compatible LLM, you’ll need:
  • Model name
  • Base URL
  • API key
Ensure your LLM:
  • Streamable (ie. via SSE)
  • Uses the /chat/completions endpoint

​
1. model

Name of the custom model you want to use. 
"model": "gpt-3.5-turbo"

​
2. base_url

Base URL of your LLM endpoint.
Do not include route extensions in the base_url.
"base_url": "https://your-llm.com/api/v1"

​
3. api_key

API key to authenticate with your LLM provider. 
"api_key": "your-api-key"
base_url and api_key are required only when using a custom model.

​
4. tools

Optionally enable tool calling by defining functions the LLM can invoke.
Please see LLM Tool Calling for more details.

​
5. speculative_inference

When set to true, the LLM begins processing speech transcriptions before user input ends, improving responsiveness. 
"speculative_inference": true
This is field is optional, but recommended for better performance.

​
6. headers

Optional headers for authenticating with your LLM. 
"headers": {
  "Authorization": "Bearer your-api-key"
}
This field is optional, depending on your LLM model.

​
7. extra_body

Add parameters to customize the LLM request, such as temperature. 
"extra_body": {
  "temperature": 0.5
}
This is field is optional.

​
Example Configuration

{
  "persona_name": "Storyteller",
  "system_prompt": "You are a storyteller who entertains people of all ages.",
  "context": "Your favorite stories include Little Red Riding Hood and The Three Little Pigs.",
  "pipeline_mode": "full",
  "default_replica_id": "r665388ec672",
  "layers": {
    "llm": {
      "model": "gpt-3.5-turbo",
      "base_url": "https://api.openai.com/v1",
      "api_key": "your-api-key",
      "speculative_inference": true
    }
  }
}
Refer to the Create Persona API for a full list of supported fields.

​
Perception

When using the raven-0 perception model with a custom LLM, your LLM will receive system messages containing visual context extracted from the user’s video input. 
{
    "role": "system",
    "content": "<user_appearance>...</user_appearance> <user_emotions>...</user_emotions> <user_screenshare>...</user_screenshare>"
}

​
Basic Perception model

If you use the Basic perception model, your LLM will receive the following user messages (instead of a system message): 
{
    "role": "user",
    "content": "USER_SPEECH: ... VISUAL_SCENE: ..."
}

​
Disabled Perception model

If you disable the perception model, your LLM will not receive any special messages.