Artificial intelligence has transformed the way we learn, but most AI learning tools still focus primarily on English. As a student, I wanted to explore how AI could make learning more accessible in Indian languages.

That idea led me to choose the Learning & Literacy track during the 10 Days of Voice Agents – VoiceForBharat Edition challenge.

My goal was simple: build an AI voice agent that allows college students to learn and discuss academic subjects in their own language instead of being limited to English-only interactions.

Meet LearnBuddy

LearnBuddy is an AI-powered voice agent designed specifically for college students.

Instead of typing long questions into a chatbot, students can simply speak naturally and receive immediate responses through voice conversations.

The agent supports multilingual interactions, allowing students to communicate in Indian languages while discussing academic topics. This approach makes learning more comfortable, more accessible, and more natural.

Voice interactions also save time. Speaking is often faster than typing, especially when students need quick explanations while studying.

By combining speech recognition, large language models, and realistic text-to-speech technology, LearnBuddy creates an interactive learning experience that feels more like a conversation than a traditional search engine.

The Technology Stack

Building a real-time voice agent requires multiple components that work together seamlessly. LearnBuddy combines speech recognition, language understanding, speech synthesis, real-time communication, and persistent memory to create an interactive learning experience.

Each component plays a specific role in the voice pipeline.

• Deepgram Nova-3 converts spoken language into text.

• Gemini 3.5 Flash Lite processes the user's request and generates responses.

• Murf Falcon converts the generated response back into natural-sounding speech.

• LiveKit handles real-time audio streaming between the user and the voice agent.

• Next.js provides the frontend interface.

• SQLite stores user information and enables memory for returning users.

How the Voice Agent Works

LearnBuddy follows a real-time voice pipeline.

1. The student speaks to the agent.

2. Deepgram Nova-3 converts the spoken audio into text.

3. Gemini 3.5 Flash Lite analyzes the request and generates a response.

4. Murf Falcon converts the generated text into natural-sounding speech.

5. LiveKit manages the real-time communication between the user and the agent.

6. SQLite stores user information when memory is enabled and the user provides consent.

This architecture allows students to have natural, voice-based conversations with the agent while receiving responses in multiple languages.

Features I Implemented

Indian Voice Powered by Murf Falcon

One of the most important goals of this project was to provide a voice experience that felt natural to Indian users. Murf Falcon made it possible to generate fast and realistic speech for the voice agent.

Multilingual Conversations

LearnBuddy supports multiple Indian languages, including:

• English

• Hindi

• Telugu

• Tamil

• Bengali

• Odia

Students can interact with the agent in the language they are most comfortable using.

Memory for Returning Users

I implemented memory using SQLite so the agent could remember user information across conversations.

To protect user privacy, I added a consent-based memory system. The agent stores information only after receiving explicit permission from the user.

Tool Calling

The agent can use external tools to retrieve information and perform useful tasks during a conversation.

Outbound Calls

I implemented outbound calling so the agent could initiate phone calls instead of waiting for users to start every interaction.

Human Escalation

An AI agent should not attempt to solve every problem on its own.

I implemented a human escalation workflow so the agent could recognize situations that required human intervention and transfer the conversation when necessary.

Call Analytics Dashboard

I built a dashboard to track call outcomes and monitor the performance of the voice agent.

Specialist Agent Handoffs

I implemented specialist-agent handoffs so the main agent could transfer conversations to another agent with more specific expertise when appropriate.

Challenges I Faced

Building a voice agent over ten days was an excellent learning experience, but it also introduced several technical challenges.

Pronunciation in Indian Languages

One of the biggest challenges was improving pronunciation in Indian languages. Some words were initially pronounced with an English accent. I experimented with different language settings and voice configurations to improve the experience.

Speech Detection

At times, the agent did not detect my speech correctly during the first attempt. After debugging the issue, I adjusted the voice activity detection parameters to improve speech recognition.

Voice Configuration

Configuring the text-to-speech engine required multiple iterations. Testing different voices and language settings helped me achieve better results.

Deployment and Environment Configuration

I also encountered deployment issues while configuring the project environment. Careful debugging of the project structure and environment variables helped me resolve those problems.

How to Set Up and Run the Project

Clone the Repository

git clone https://github.com/buildWithPooja/murf-livekit-starter.git

Install Dependencies

cd murf-livekit-starter

pnpm install

uv sync

Configure Environment Variables

Store all API keys in environment files.

Example:

backend/.env
frontend/.env.local

Never expose API keys, phone numbers, or user data in a public repository.

Run the Backend

uv run src

Run the Frontend

pnpm dev

Testing the Agent

After starting the backend and frontend, open the local application in your browser and connect to the agent.

Test the following features:

• Multilingual conversations

• Memory

• Tool calling

• Outbound calls

• Human escalation

• Specialist-agent handoffs

Screenshots and Demonstrations

The screenshots in this article demonstrate the different stages of the development process, including the user interface, analytics dashboard, outbound calling, human escalation, and specialist-agent handoffs.

GitHub Repository

https://github.com/buildWithPooja/murf-livekit-starter

Final Thoughts

The 10 Days of Voice Agents challenge provided an excellent opportunity to explore the complete lifecycle of building a real-time voice agent.

Over ten days, I learned how to combine speech recognition, large language models, text-to-speech systems, memory, tool calling, analytics, and human-in-the-loop workflows into a single application.

Voice AI is evolving rapidly, and projects like LearnBuddy demonstrate how these technologies can make learning more accessible for students.

I hope this tutorial helps other developers build their own multilingual voice agents and continue exploring the possibilities of Voice AI.