We Gave OpenClaw a Voice in 24 Hours: Here’s What Happened

The Problem: Your AI Voice Assistant Should Actually Have a Voice

OpenClaw handles text like a champ. You type, it responds with context-aware answers. But what about when you’re driving? Or cooking dinner with flour-covered hands? That’s when you really need an AI voice assistant that just works.

So we gave OpenClaw the ability to answer phone calls. Not some clunky “press 1 for sales” nightmare, but a real conversational AI that recognizes your voice, answers questions, and actually sounds human.

Here’s what we learned building a voice AI tutorial from scratch.

What This AI Voice Assistant Can Do

Our family phone bot handles:

• •Caller ID recognition – Greets family members by name when they call
• •Real-time answers – Weather, web searches, local business info via OpenClaw’s tools
• •Outbound calling – “Call [contact] and ask when they’re free” actually works
• •Natural hang-ups – No awkward silences or robotic “GOODBYE”
• •Fast responses – 2-3 seconds using Claude Haiku for conversational AI speed

It’s not perfect. But it’s real, and that matters.

How We Built This Voice AI Tutorial: The Architecture

After several false starts, here’s the stack that actually worked:

┌─────────────────────────────────────────────┐
│  Twilio (Your phone number)                 │
│  • Phone calls & routing                    │
│  • Speech-to-text transcription             │
│  • Text-to-speech (neural voices)           │
└──────────────────┬──────────────────────────┘
                   │ Webhooks
                   ↓
┌─────────────────────────────────────────────┐
│  Cloudflare Workers (Edge compute)          │
│  • Zero cold-start responses                │
│  • Returns TwiML instantly                  │
│  • Triggers async AI processing             │
└──────────────────┬──────────────────────────┘
                   │ Async HTTP
                   ↓
┌─────────────────────────────────────────────┐
│  DigitalOcean Server (Node.js)              │
│  • Pattern pre-checking                     │
│  • Outbound call handling                   │
│  • Coordinates OpenClaw queries             │
└──────────────────┬──────────────────────────┘
                   │ HTTP API
                   ↓
┌─────────────────────────────────────────────┐
│  OpenClaw Gateway (Local AI)                │
│  • Claude Haiku 4-5 for speed               │
│  • Weather, search, memory tools            │
│  • Persistent conversation context          │
└─────────────────────────────────────────────┘

Why This Tech Stack for Our AI Voice Assistant?

Twilio AI – Handles the gnarly telephony stuff (SIP protocols, audio codecs, transcription). You just send XML responses.

Cloudflare Workers – Edge deployment means zero cold starts. Twilio kills calls after 15 seconds of silence. Workers guarantee instant replies.

DigitalOcean – OpenClaw needs Node.js, file system access, and persistent memory. You can’t run that in a serverless function.

OpenClaw – The brain of this voice AI tutorial. Already had weather, web search, and memory built in. We just wired it to a phone number.

The Challenges of Building a Conversational AI Phone Bot

Challenge #1: Twilio’s 15-Second Timeout Will Kill Your Call

When someone speaks, Twilio transcribes it and pings your webhook. You have 15 seconds to respond or the call drops dead. OpenClaw can take 5-10 seconds to think. Add network latency and you’re toast.

Our fix: The Cloudflare Worker instantly returns holding TwiML (“Hang on, checking that…”), then triggers the DO server asynchronously. That server processes the query, stores results in Cloudflare KV, and redirects Twilio to grab the cached response. Total user wait: 3-5 seconds. Twilio never times out.

Challenge #2: Phone Conversations Aren’t Chat Windows

You can’t scroll back through a phone call. You can’t read 10 options. And if your AI voice assistant rattles off a phone number at normal speed, good luck writing it down.

VOICE AI RULES:
- Keep responses under 40 words
- Never list more than 3 items
- For numbers/addresses, offer to text instead
- Use natural speech patterns (no "Here is a list:")
- Include [HANGUP] when ending the conversation

Claude Haiku actually follows these instructions. Shocking.

Challenge #3: Speed vs. Quality in Conversational AI

First attempt: Claude Opus. Beautifully nuanced responses. Also 10+ seconds per turn, which feels like an eternity on a phone call.

Our fix: Switched entirely to Claude Haiku. Response time dropped to 2-3 seconds. Quality loss? Almost none. For short voice exchanges, Haiku is plenty smart.

We also pre-check common patterns before calling OpenClaw:

• •“Call [number] and [ask something]” → Handle with Twilio API directly
• •“What time does [place] close?” → Hit Google Places API
• •“Goodbye” / “Thanks” → Random farewell + hang up

Only complex queries hit OpenClaw. This saves 1-2 seconds on routine stuff.

Challenge #4: Twilio URL Length Limits Are Sneaky

First attempt: pass AI responses via URL parameters like /voice/callback?text=Hello+how+can+I+help...

Works great until the response is long. Then Twilio returns a 502 error because URLs cap at ~2000 characters.

Our fix: Store responses in Cloudflare KV with a unique ID, pass just the ID: /voice/callback?responseId=abc123. The Worker fetches the full response from KV and converts it to TwiML.

Challenge #5: Outbound Calling is Surprisingly Weird

We wanted: “Call my contact and ask what time we’re meeting.” What we got: A rabbit hole of Twilio Conference rooms, hold music, and bridge connections.

Here’s how outbound calling actually works:

1. Put the original caller into a Twilio Conference with hold music
2. Initiate a new outbound call
3. Bridge that call into the conference
4. Let the AI voice assistant talk to the recipient
5. Summarize the conversation
6. Redirect the original caller out with the summary

Took us an entire afternoon. The key was endConferenceOnExit=false so hold music doesn’t cut out mid-call.

What Actually Surprised Us About Building This Phone Bot

The Good Stuff

Neural voices sound shockingly human. Started with standard Polly voices (robotic, flat). Switched to Polly Neural (Polly.Brian-Neural) and suddenly it felt like talking to a real assistant, not a phone bot.

Claude Haiku is fast enough. 2-3 seconds feels natural in conversation. Humans pause to think too. Haiku’s pauses feel… right.

People actually use it. We gave the number to friends. They call for weather, business hours, and jokes. (The jokes are terrible, but the AI voice assistant tries.)

The Weird Stuff

Hang-up detection is black magic. Tried regex patterns: /(bye|goodbye|thanks|see you)/i. Worked 80% of the time. The other 20%? People say “Okay, bye!” and Claude keeps talking. Or they say “Thanks for that info” mid-conversation and it hangs up.

We ended up asking Claude to decide: “Does this sound like the user wants to end the call? YES or NO.” Works better, adds 200ms of latency.

Twilio transcription has… quirks. “What’s the weather?” works perfectly. “What time does Chick-fil-A close?” becomes “what time does Chick Filet close.” We pass the mangled transcription to Claude, which is surprisingly good at guessing intent.

KV storage is eventually consistent. We store responses in Cloudflare KV, then immediately redirect Twilio to fetch them. 99% of the time, perfect. 1% of the time, the write hasn’t propagated and the Worker returns “No response available.” Added retries with exponential backoff. Problem solved.

Performance & Cost of This AI Voice Assistant

Speed

Cost Breakdown

Total per call: About $0.02-0.05 per 2-minute call. Cheaper than a landline.

Important: Voice support works immediately after purchasing a Twilio number. Outbound SMS requires A2P (Application-to-Person) registration, which takes 1-2 weeks for carrier approval. You can start with voice-only and add texting later.

Code Samples from This Voice AI Tutorial

Cloudflare Worker (Edge TwiML Response)

// Handles incoming calls to our AI voice assistant
async function handleInboundCall(request, env) {
  const formData = await request.formData();
  const from = formData.get('From');
  const callSid = formData.get('CallSid');
  // Recognized caller? Skip IVR menu
  if (from === env.ADMIN_CELL) {
    return new Response(`<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Say voice="Polly.Brian-Neural">Hello! How can I help you?</Say>
  <Gather input="speech" action="${host}/voice/response?authenticated=true" />
</Response>`, {
      headers: { 'Content-Type': 'application/xml' }
    });
  }
  // Unknown caller? Friendly IVR menu
  return new Response(`<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Gather action="${host}/voice/gather" numDigits="1">
    <Say voice="Polly.Brian-Neural">
      Thank you for calling.
      For assistance, press 1.
      Or stay on the line for an agent.
    </Say>
  </Gather>
</Response>`, {
    headers: { 'Content-Type': 'application/xml' }
  });
}

DigitalOcean Server (OpenClaw Integration for Phone Bot)

// Call OpenClaw via HTTP for conversational AI
async function askOpenClaw(callSid, message, isAuthenticated, from) {
  const response = await fetch(`${OPENCLAW_URL}/v1/chat/completions`, {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${OPENCLAW_TOKEN}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      model: 'anthropic/claude-haiku-4-5',
      user: `phone-${callSid}`, // Session isolation per call
      messages: [
        {
          role: 'system',
          content: `You are a phone assistant. Keep responses under 40 words. 
                    You have access to weather, web search, and local business tools. 
                    To hang up, include [HANGUP] in your response.`
        },
        {
          role: 'user',
          content: message
        }
      ]
    })
  });
  const data = await response.json();
  return data.choices[0].message.content;
}

Key Lessons from Building This Voice AI Tutorial

Architecture Choices Matter More Than You Think

First try: run everything in Cloudflare Workers. Hit the 50ms CPU limit immediately. Moved AI processing to DigitalOcean. Fixed.

Second try: call OpenClaw directly from the Worker. Network lag + AI processing = Twilio timeout. Added KV-based async pattern. Fixed.

Takeaway: Design for constraints (Twilio’s 15s timeout, Workers’ CPU limits). Don’t fight the platform.

Voice UX is a Different Beast

Text chat works with lists, links, and paragraphs. Voice doesn’t. We spent hours tweaking the system prompt to get natural-sounding responses:

• •“It’s 72 degrees and sunny” (not “The current temperature is 72°F with clear skies and visibility of 10 miles”)
• •“I can text you that” (not reading a 10-digit phone number aloud)
• •“Let me check on that” (buys time for async processing)

Takeaway: Design for the medium. Phone conversations are sequential, audio-only, and time-sensitive.

Conversational AI is Production-Ready

We were nervous about putting an LLM on a public phone number. What if it hallucinates? What if it says something weird?

Two months later: zero disasters. Claude Haiku is reliable. It follows instructions. When it doesn’t know something, it says so.

Takeaway: LLMs work in production if you:

1. Set clear constraints (response length, format, behavior)
2. Pre-check patterns before hitting the LLM (saves money + reduces errors)
3. Monitor logs and iterate based on real usage

What We’re Building Next for This AI Voice Assistant

Coming Soon

• •Voicemail transcription — If the phone bot doesn’t answer (server down, etc.), transcribe voicemail and send via SMS.
• •Proactive calling — “Call me at 3pm to remind me about the dentist.”
• •Multi-party conferencing — “Call multiple people so we can all talk.”
• •Voice cloning — Use ElevenLabs or PlayHT to clone your actual voice for the AI voice assistant. Creepy? Maybe. Cool? Definitely.

What We’d Do Differently

• •Start with Claude Haiku from day one. We wasted time optimizing Opus when the real solution was using a faster model for conversational AI.
• •Use Twilio Media Streams. We currently use TwiML (request/response). Twilio also supports WebSocket-based Media Streams for real-time audio. Lower latency, but way more complex. Worth exploring for v2 of this voice AI tutorial.
• •Add SMS fallback automatically. For any response over 50 words, the phone bot should say “I’ll text you the details” and send an SMS. We do this manually now, should be automatic.

Getting Your Own Phone Number

You can purchase a Twilio phone number and set up your own OpenClaw voice assistant in about 5 minutes.

Twilio Setup (Quick Start)

1. Create account at twilio.com/try-twilio (free trial includes $15 credit)
2. Buy a phone number: Phone Numbers → Buy a Number — Choose a local number in your area code (~$1/month), voice-capable numbers work immediately
3. Configure webhooks: Voice webhook: https://your-worker.workers.dev/voice/inbound — Save your Account SID and Auth Token for later
4. Start taking calls — Voice support works instantly, no extra setup

Voice vs. SMS: What Works Out of the Box?

Recommendation: Start with voice-only, add SMS later if needed. The voice assistant is fully functional without texting capabilities.

Try Building Your Own AI Voice Assistant with OpenClaw

What You’ll Need

• •OpenClaw — Local AI assistant framework
• •Twilio account — Voice works instantly, SMS needs A2P registration (~$1/month phone number)
• •Cloudflare account — Workers free tier works fine
• •DigitalOcean droplet — $12/month (2GB RAM) or any Linux server
• •Anthropic API key — For Claude Haiku

The Honest Bottom Line

Building an AI voice assistant in 2026 is surprisingly achievable. The tools exist. Claude Haiku is fast enough. The cost is negligible.

What surprised us most? How normal it feels. After a week of using our phone bot, calling it feels as natural as texting. It just works.

Is it perfect? Nope. Does it occasionally misunderstand you? Yep. But so do humans.

The future of conversational AI isn’t just chatbots on screens. It’s systems that meet you where you are—text, voice, whatever. We gave OpenClaw a voice. What will you build?

—The CAVENDO Team, February 2026