Skip to main content

Runbook: Voice Platform Troubleshooting

Overview​

  • What: Troubleshoot voice/telephony platform issues
  • When: Calls not connecting, audio issues, STT/TTS failures
  • Duration: 10-30 minutes
  • Services: voice-api, voice-agent, voice-frontend, telephony, livekit

Prerequisites​

Quick Health Check​

ssh ravenhelm@100.115.101.81

# Check voice containers
docker ps | grep -E 'voice|livekit|telephony'

# Quick API health checks
curl -s https://voice.ravenhelm.dev/health | jq
curl -s https://livekit.ravenhelm.dev/health 2>/dev/null || echo 'Check LiveKit directly'

Common Issues​

Issue 1: Calls Not Connecting​

Symptoms: Inbound calls fail, Twilio webhook errors

# Check telephony service
docker logs --tail 100 telephony

# Check for webhook errors
docker logs telephony 2>&1 | grep -i 'webhook\|twilio\|error' | tail -20

# Verify Twilio webhook URL is correct
# Should point to: https://telephony.ravenhelm.dev/webhook

# Test webhook endpoint
curl -s -X POST https://telephony.ravenhelm.dev/webhook -d 'test=1'

Issue 2: No Audio / One-Way Audio​

Symptoms: Call connects but no audio in one or both directions

# Check LiveKit status
docker logs --tail 100 livekit

# Check for ICE/STUN/TURN issues
docker logs livekit 2>&1 | grep -i 'ice\|stun\|turn\|candidate' | tail -20

# Verify LiveKit ports are open
# UDP 7881 (WebRTC)
# TCP 7880 (API)

# Check Coturn (TURN server) if configured
docker ps | grep coturn

Issue 3: Speech-to-Text (STT) Failures​

Symptoms: Agent not understanding speech, transcription errors

# Check voice-api for STT errors
docker logs voice-api 2>&1 | grep -i 'whisper\|transcri\|stt\|speech' | tail -20

# Verify Whisper is running
docker ps | grep whisper

# Test Whisper directly
curl -s http://localhost:9000/health

# Check Whisper GPU utilization (if using GPU)
nvidia-smi 2>/dev/null || echo 'No GPU'

Issue 4: Text-to-Speech (TTS) Failures​

Symptoms: Agent not speaking, audio playback issues

# Check for TTS errors
docker logs voice-api 2>&1 | grep -i 'piper\|tts\|speech\|audio' | tail -20

# Verify Piper is running
docker ps | grep piper

# Test Piper directly
curl -s http://localhost:5000/health 2>/dev/null || echo 'Check Piper port'

Issue 5: High Latency / Choppy Audio​

Symptoms: Delayed responses, audio cutting out

# Check container resources
docker stats --no-stream voice-api voice-agent livekit

# Check network latency
ping -c 5 livekit.ravenhelm.dev

# Check for buffer underruns in logs
docker logs voice-api 2>&1 | grep -i 'buffer\|latency\|delay' | tail -20

# Check LiveKit room stats
curl -s https://livekit.ravenhelm.dev/api/rooms | jq

Issue 6: WebSocket Connection Issues​

Symptoms: Frontend can't connect, real-time updates failing

# Check voice-frontend
docker logs --tail 100 voice-frontend

# Check for WebSocket errors
docker logs voice-api 2>&1 | grep -i 'websocket\|ws\|connection' | tail -20

# Verify Traefik WebSocket headers
curl -sI https://voice.ravenhelm.dev | grep -i upgrade

Full Service Restart​

cd ~/ravenhelm/services/voice

# Graceful restart
docker compose restart

# Full recreate (if issues persist)
docker compose down
docker compose up -d

# Also restart LiveKit if audio issues persist
cd ~/ravenhelm/services/livekit
docker compose restart

# Watch logs
docker compose logs -f --tail 50

Verification​

# Test voice API health
curl -s https://voice.ravenhelm.dev/health | jq

# Test telephony webhook
curl -s -X POST https://telephony.ravenhelm.dev/health

# Make a test call (via Twilio console or phone)
# Monitor logs during call:
docker logs -f voice-agent telephony

Twilio Troubleshooting​

# Check Twilio webhook configuration
# 1. Log into Twilio Console
# 2. Go to Phone Numbers > Manage > Active Numbers
# 3. Verify webhook URL: https://telephony.ravenhelm.dev/webhook
# 4. Check webhook logs in Twilio Console > Monitor > Logs

# Test Twilio connectivity
twilio api:core:calls:list --limit 5

Escalation​

If issues persist:

  1. Check Twilio service status: https://status.twilio.com
  2. Review LiveKit logs for room/participant issues
  3. Check if Whisper/Piper models need updating
  4. Review recent changes in GitLab: https://gitlab.ravenhelm.dev/voice/saaa