Voice Agents

Add realtime speech to your Newnadoon agents with NewnaVoiceKit.

Your agent still uses the same workflow logic as text chat. NewnaVoiceKit handles the microphone session, speech-to-text, text-to-speech, and turn-taking. You configure personality and voice providers in TypeScript with createVoiceAgent.

This page walks through the sample Voice Assistant that ships with npm create newnadoon and npx newnadoon-voice setup.

Requires @newnadoon/ai-framework 1.0.18 or later.

What you will do

1. Install voice runtime dependencies once.
2. Fill in API keys in .env.
3. Run the voice worker and the debug dashboard.
4. Talk to (or type to) the sample Voice Assistant and see the Execution Timeline update.

1. Prerequisites

You need:

• A Newnadoon project (npm create newnadoon@latest, or an existing app with @newnadoon/ai-framework)
• An OpenAI API key (the sample uses useOpenAIModel)
• Postgres running if your sample agent uses PostgresStorageAdapter (the default Voice Assistant does)
• NewnaVoiceKit session credentials (media room)
• A Deepgram API key (speech-to-text)
• A Cartesia API key (text-to-speech)

Registry reminder (same as Get started):

@newnadoon:registry=https://npm.newnadoon.com/

Upgrade the framework if needed:

$ npm install @newnadoon/ai-framework@latest

2. One-time voice setup

From your project root:

$ npx newnadoon-voice setup

Then download voice model files used by the worker:

$ npx newnadoon-voice download-files

setup will:

• install voice runtime packages (including livekit-client)
• create lib/agents/voice-assistant.ts if it is missing
• register voice-assistant in newnadoon.json with a voice: {} capability marker
• append NewnaVoiceKit env keys to .env.example when that file exists

New projects from create-newnadoon (≥ 0.2.12) already include the Voice Assistant sample. You still need setup for the runtime dependencies.

3. Configure environment variables

Add these to your project .env (or .env.local):

# LLM (same as other sample agents)
OPENAI_API_KEY=sk-...
OPENAI_MODEL=gpt-4o-mini
 
# Postgres (sample Voice Assistant uses PostgresStorageAdapter)
POSTGRES_URL=postgresql://postgres:postgres@localhost:5432/newnadoon
# or DATABASE_URL=...
 
# NewnaVoiceKit session (media plane)
NEWNAVOICEKIT_API_KEY=your_api_key
NEWNAVOICEKIT_API_SECRET=your_api_secret
# Optional — defaults to hosted NewnaVoiceKit
# NEWNAVOICEKIT_URL=wss://livekit.newnadoon.io
NEWNAVOICEKIT_AGENT_NAME=newnavoicekit
 
# Speech providers (BYO keys)
NEWNAVOICEKIT_STT_API_KEY=your_deepgram_key
NEWNAVOICEKIT_TTS_API_KEY=your_cartesia_key
# Optional Cartesia voice id
NEWNAVOICEKIT_VOICE_ID=
 
# Default agent id when dispatch metadata omits one
NEWNAVOICEKIT_DEFAULT_AGENT_ID=voice-assistant
 
# Required for Execution Timeline updates in the debug dashboard
NEWNAVOICEKIT_AGENT_RUN_URL=http://localhost:2024/api/voice/turn

What each voice block does:

Without NEWNAVOICEKIT_AGENT_RUN_URL, the worker can still speak answers, but the dashboard Execution Timeline will stay empty.

4. The sample Voice Assistant

After setup (or scaffolding), you should have:

lib/agents/voice-assistant.ts

This file builds a small chat workflow, then wraps it with createVoiceAgent:

import 'dotenv/config'
import {
  createVoiceAgent,
  createWorkflowBuilder,
  createStep,
  useOpenAIModel,
  PostgresStorageAdapter,
} from '@newnadoon/ai-framework'
 
export default async function createVoiceAssistant() {
  const llm = await useOpenAIModel(process.env.OPENAI_MODEL || 'gpt-4o-mini')
 
  const chat = createWorkflowBuilder({
    storageAdapter: PostgresStorageAdapter({
      connectionString:
        process.env.POSTGRES_URL ||
        process.env.DATABASE_URL ||
        'postgresql://postgres:postgres@localhost:5432/newnadoon',
    }),
  })
    .addStep('chat', createStep({ llm }))
    .setEntryPoint('chat')
    .build()
 
  return createVoiceAgent({
    agent: chat,
    instructions:
      'You are a helpful voice assistant. Keep answers short, clear, and conversational. Avoid markdown.',
    greeting: 'Hi, I am your Newnadoon voice assistant. How can I help you today?',
    stt: {
      provider: 'deepgram',
      model: 'nova-3',
    },
    tts: {
      provider: 'cartesia',
      model: 'sonic-3',
      voiceId: process.env.NEWNAVOICEKIT_VOICE_ID,
    },
  })
}

How to read this file:

1. createWorkflowBuilder + createStep({ llm }) — normal Newnadoon chat agent (same pattern as text agents).
2. createVoiceAgent({ agent, instructions, greeting, stt, tts }) — marks the workflow as voice-capable and attaches STT/TTS settings the worker reads at runtime.
3. instructions / greeting — change these to customize personality and the first spoken line.
4. stt / tts — Deepgram and Cartesia by default; keys come from .env unless you pass apiKey inline.

newnadoon.json entry

{
  "agents": {
    "voice-assistant": {
      "path": "@agents/voice-assistant",
      "name": "Voice Assistant",
      "description": "NewnaVoiceKit voice agent — conversational assistant with STT/TTS.",
      "voice": {}
    }
  },
  "env": ".env"
}

The empty "voice": {} object is a capability marker. It tells the debug dashboard to show the Voice panel for this agent. Instructions and provider settings live in createVoiceAgent(...), not in JSON.

5. Run voice locally (two terminals)

Use one voice worker for this project. Do not run a second newnadoon-voice from another repo at the same time — both register as newnavoicekit and sessions can go to the wrong worker (voice answers, timeline stays empty).

Terminal A — voice worker

$ npx newnadoon-voice dev

You should see your project path and:

Turn API: http://localhost:2024/api/voice/turn

Keep this process running.

Terminal B — debug dashboard

$ npx newnadoon dev

Open:

 http://localhost:2024

6. Try the sample in the debugger

1. In the dashboard, select [voice] Voice Assistant.
2. Click Connect voice (the mic enables automatically after you allow browser permission).
3. Speak a short question, for example: “What is 4 plus 4?”
4. Or type the same question and click Send turn (useful when you cannot use a mic).

What you should see:

• The Voice transcript updates with user and assistant lines.
• The agent speaks the reply.
• The Execution Timeline shows a new turn with a Final Answer.
• The Initial State JSON task updates to your spoken or typed question.

How a voice turn works

1. Final user transcript becomes the agent task.
2. With NEWNAVOICEKIT_AGENT_RUN_URL set, the worker (or Send turn) calls /api/voice/turn.
3. The dashboard runs your agent with the same debug wrapping as Start Run.
4. The worker speaks the returned reply with Cartesia TTS.

Text Start Run still works for this agent — it runs the chat workflow without the microphone.

7. Customize the sample

Edit lib/agents/voice-assistant.ts, then restart npx newnadoon-voice dev so the worker reloads greeting and instructions.

Common changes:

Optional JSON overrides: you can put instructions, greeting, stt, or tts under the voice block in newnadoon.json. Precedence is:

1. createVoiceAgent(...) in code
2. newnadoon.json voice
3. Framework defaults

8. Add voice to your own frontend (optional)

The debugger Voice panel is enough for local testing. For your own UI, use the React hook:

'use client'
 
import { useNewnaVoice } from '@newnadoon/ai-framework/react'
 
export function VoicePanel() {
  const {
    status,
    error,
    micEnabled,
    transcripts,
    connect,
    disconnect,
    toggleMic,
  } = useNewnaVoice({
    agentId: 'voice-assistant',
    tokenUrl: '/api/newnadoon/voice/token',
  })
 
  return (
    <div>
      <p>Status: {status}</p>
      {error && <p>{error}</p>}
      <button onClick={() => connect()}>Connect</button>
      <button onClick={() => toggleMic()}>
        {micEnabled ? 'Mute' : 'Unmute'}
      </button>
      <button onClick={() => disconnect()}>Disconnect</button>
      <ul>
        {transcripts.map((t, i) => (
          <li key={i}>
            <strong>{t.speaker}:</strong> {t.text}
          </li>
        ))}
      </ul>
    </div>
  )
}

Token routes:

• Next.js App Router with createNewnadoonApiRoute(): GET /api/newnadoon/voice/token?agentId=voice-assistant
• Express via createExpressApp(): GET /api/voice/token?agentId=voice-assistant
• Debug dashboard: GET /api/voice/token?agentId=voice-assistant

Your app must depend on livekit-client (newnadoon-voice setup installs it).

9. Troubleshooting

Quick API check after a turn (replace the workflow id from the dashboard dropdown):

curl http://localhost:2024/api/workflows/<workflowId>/runs

• 0 runs → the turn never hit /api/voice/turn (env or wrong worker).
• Runs present → data is saved; refresh or re-select the workflow in the UI.

Related pages

• Get started
• Debugger visibility dashboard
• CLI
• Examples
• Bring agents to frontend apps