Health Consultation

This application demonstrates a simple videocall with a Health Care Consultant persona. The application uses Tavus CVI to create an interactive AI host that responds to user inputs and performs actions through tool calls.

Prerequisites

Ensure that you have installed:

Node.js

Getting Started

Clone the Sample Project

git clone https://github.com/Tavus-Engineering/tavus-examples.git

Run the Sample Project

Navigate to examples > health-consultation-with-tavus folder.
Install modules:

npm i

Run the project:

npm run start

Navigate to the localhost:3000/health-consultation-with-tavus/.
Enter your name and click General Health or Skin & Dermatology.

Tool Call Flow

Here’s a quick overview of how tool calls work in this Health Consultation app:

User Command Detected

The user says What is the cure to X ?to the replica. The application listens for messages using:

call.on('app-message', handleAppMessage);

Tool Call Detection

The replica triggers the get_cures tool:

if (message.message_type === 'conversation' && message.event_type === 'conversation.tool_call') {
  const toolCall = message.properties;
  
    if (toolCall.name === 'get_cures') {
        try {
          const args = JSON.parse(toolCall.arguments);
          const disease = args.disease;
          console.log('User wanted to know cures for', disease);
          
          const diseaseclear = disease.trim().toLowerCase();
          const cure = cureFor[diseaseclear];
          
          const responseMessage = {
            message_type: "conversation",
            event_type: "conversation.echo",
            conversation_id: message.conversation_id,
            properties: {
              text: `The cure for ${disease} is ${cure}.`
            }
          };

          console.log('Sending echo message:', responseMessage);
          
          // Use the call object to send app message
          const call = callRef.current;
          if (call && typeof call.sendAppMessage === 'function') {
            call.sendAppMessage(responseMessage, '*');
            console.log('Message sent successfully');
          } else {
            console.error('Call object is not available or sendAppMessage method is missing');
          }
        } catch (error) {
          console.error('Error in processing cure request:', error);
        }
      }
}

The app then try to search the cure based on a defined dictionary:

const cureFor = {
  "cold": "Paracetamol",
  "flu": "Oseltamivir",
  "asthma": "Salbutamol",
  "migraine": "Sumatriptan",
  "depression": "Fluoxetine",
  "epilepsy": "Sodium Valproate"
};

const diseaseclear = disease.trim().toLowerCase();
const cure = cureFor[diseaseclear];

Application Responds

Once the result is generated, the app constructs and sends an echo message back to the replica:

const responseMessage = {
  message_type: "conversation",
  event_type: "conversation.echo",
  conversation_id: message.conversation_id,
  properties: {
    text: `The cure for ${disease} is ${cure}.`
  }
};

call.sendAppMessage(responseMessage, '*');

The replica receives the echo message and replies: The cure for cold is Paracetamol.

User Command Detected

The user says What is the cure to X ?to the replica. The application listens for messages using:

call.on('app-message', handleAppMessage);

Tool Call Detection

The replica triggers the get_cures tool:

if (message.message_type === 'conversation' && message.event_type === 'conversation.tool_call') {
  const toolCall = message.properties;
  
    if (toolCall.name === 'get_cures') {
        try {
          const args = JSON.parse(toolCall.arguments);
          const disease = args.disease;
          console.log('User wanted to know cures for', disease);
          
          const diseaseclear = disease.trim().toLowerCase();
          const cure = cureFor[diseaseclear];
          
          const responseMessage = {
            message_type: "conversation",
            event_type: "conversation.echo",
            conversation_id: message.conversation_id,
            properties: {
              text: `The cure for ${disease} is ${cure}.`
            }
          };

          console.log('Sending echo message:', responseMessage);
          
          // Use the call object to send app message
          const call = callRef.current;
          if (call && typeof call.sendAppMessage === 'function') {
            call.sendAppMessage(responseMessage, '*');
            console.log('Message sent successfully');
          } else {
            console.error('Call object is not available or sendAppMessage method is missing');
          }
        } catch (error) {
          console.error('Error in processing cure request:', error);
        }
      }
}

The app then try to search the cure based on a defined dictionary:

const cureFor = {
  "cold": "Paracetamol",
  "flu": "Oseltamivir",
  "asthma": "Salbutamol",
  "migraine": "Sumatriptan",
  "depression": "Fluoxetine",
  "epilepsy": "Sodium Valproate"
};

const diseaseclear = disease.trim().toLowerCase();
const cure = cureFor[diseaseclear];

Application Responds

Once the result is generated, the app constructs and sends an echo message back to the replica:

const responseMessage = {
  message_type: "conversation",
  event_type: "conversation.echo",
  conversation_id: message.conversation_id,
  properties: {
    text: `The cure for ${disease} is ${cure}.`
  }
};

call.sendAppMessage(responseMessage, '*');

The replica receives the echo message and replies: The cure for cold is Paracetamol.

User Command Detected

The user says What is the cure to X ? or What is the solution to X ? to the replica. The application listens for messages using:

call.on('app-message', handleAppMessage);

Tool Call Detection

The replica triggers the get_skin_cures tool:

LLM Tool

if (message.message_type === 'conversation' && message.event_type === 'conversation.tool_call') {
  const toolCall = message.properties;
  
    if (toolCall.name === 'get_skin_cures') {
        try {
          const args = JSON.parse(toolCall.arguments);
          const disease = args.disease;
          console.log('User wanted to know cures for', disease);
          
          const diseaseclear = disease.trim().toLowerCase();
          const cure = cureForSkin[diseaseclear];
          
          const responseMessage = {
            message_type: "conversation",
            event_type: "conversation.echo",
            conversation_id: message.conversation_id,
            properties: {
              text: `If you're getting ${disease} i suggest to ${cure}.`
            }
          };
        }
}

Cures are retrieved from a predefined dictionary:

const cureForSkin = {
    "pimples": "Use a mild cleanser, avoid touching your face, and apply a benzoyl peroxide cream",
    "sunburn": "Apply aloe vera or a cooling moisturizer, and stay out of the sun",
    "wrinkles": "Use sunscreen daily, moisturize, and avoid smoking or tanning",
    "oilyskin": "Use oil-free products, gentle cleansers, and don't overwash your face",
    "darkspot": "Try topical treatments like vitamin C, retinoids, or consult for chemical peels",
    "dryskin": "Moisturize regularly, use gentle cleansers, and avoid hot showers"
};

const diseaseclear = disease.trim().toLowerCase();
const cure = cureForSkin[diseaseclear];

Application Responds

Once the result is generated, the app constructs and sends an echo message back to the replica:

const responseMessage = {
  message_type: "conversation",
  event_type: "conversation.echo",
  conversation_id: message.conversation_id,
  properties: {
    text: `The cure for ${disease} is ${cure}.`
  }
};

call.sendAppMessage(responseMessage, '*');

The replica receives the echo message and replies: The cure for pimples is Use a mild cleanser, avoid touching your face, and apply a benzoyl peroxide cream.

User Command Detected

The user says What is the cure to X ? or What is the solution to X ? to the replica. The application listens for messages using:

call.on('app-message', handleAppMessage);

Tool Call Detection

The replica triggers the get_skin_cures tool:

LLM Tool

if (message.message_type === 'conversation' && message.event_type === 'conversation.tool_call') {
  const toolCall = message.properties;
  
    if (toolCall.name === 'get_skin_cures') {
        try {
          const args = JSON.parse(toolCall.arguments);
          const disease = args.disease;
          console.log('User wanted to know cures for', disease);
          
          const diseaseclear = disease.trim().toLowerCase();
          const cure = cureForSkin[diseaseclear];
          
          const responseMessage = {
            message_type: "conversation",
            event_type: "conversation.echo",
            conversation_id: message.conversation_id,
            properties: {
              text: `If you're getting ${disease} i suggest to ${cure}.`
            }
          };
        }
}

Cures are retrieved from a predefined dictionary:

const cureForSkin = {
    "pimples": "Use a mild cleanser, avoid touching your face, and apply a benzoyl peroxide cream",
    "sunburn": "Apply aloe vera or a cooling moisturizer, and stay out of the sun",
    "wrinkles": "Use sunscreen daily, moisturize, and avoid smoking or tanning",
    "oilyskin": "Use oil-free products, gentle cleansers, and don't overwash your face",
    "darkspot": "Try topical treatments like vitamin C, retinoids, or consult for chemical peels",
    "dryskin": "Moisturize regularly, use gentle cleansers, and avoid hot showers"
};

const diseaseclear = disease.trim().toLowerCase();
const cure = cureForSkin[diseaseclear];

Application Responds

Once the result is generated, the app constructs and sends an echo message back to the replica:

const responseMessage = {
  message_type: "conversation",
  event_type: "conversation.echo",
  conversation_id: message.conversation_id,
  properties: {
    text: `The cure for ${disease} is ${cure}.`
  }
};

call.sendAppMessage(responseMessage, '*');

The replica receives the echo message and replies: The cure for pimples is Use a mild cleanser, avoid touching your face, and apply a benzoyl peroxide cream.

Replica Triggers Tool Call

If the replica visually detects acne on the participant’s face, it triggers the acne_detected perception tool:

Perception Tool

if (message.message_type === 'conversation' && message.event_type === 'conversation.perception_tool_call') {
    const perceptionToolCall = message.properties;
  
    if (perceptionToolCall.name === 'acne_detected') {
        const currentTime = Date.now();
}

Application Responds

The app constructs and sends an echo message back to the replica:

const responseMessage = {
    message_type: "conversation",
    event_type: "conversation.echo",
    conversation_id: message.conversation_id,
    properties: {
        text: `I notice that you have acne on your face. I suggest using topical antibiotics like clindamycin and erythromycin.`
        }
    };

 console.log('Sending echo message:', responseMessage);

The replica sends a helpful suggestion: I notice that you have acne on your face. I suggest using topical antibiotics like clindamycin and erythromycin.

Getting Started

Conversational Video Interface

Replica

Video Generation

Lipsync

Resources

Health Consultation

Prerequisites

Getting Started

Tool Call Flow

Getting Started

Conversational Video Interface

Replica

Video Generation

Lipsync

Resources

​Prerequisites

​Getting Started

​Tool Call Flow

Prerequisites

Getting Started

Tool Call Flow