Norns Telephony

Outbound and inbound phone calling capabilities for Norns via Twilio and Pipecat.

Last updated: 2026-01-04
Status: ✅ Production

---

Overview

Norns can make and receive phone calls with full voice AI capabilities. The system supports two distinct modes:

Mode	Description	Use Case
Scripted	Deliver a message and hang up	Notifications, reminders, simple announcements
Interactive	Full bidirectional conversation	Task delegation, information gathering, complex requests

Architecture:

┌─────────────────────────────────────────────────────────┐
│                    TELEPHONY FLOW                        │
├─────────────────────────────────────────────────────────┤
│                                                          │
│  Outbound Request                    Inbound Call       │
│       │                                    │             │
│       ▼                                    ▼             │
│  ┌──────────────────────────────────────────────┐       │
│  │         Twilio Phone Network                 │       │
│  │        (+1 737-214-3330)                     │       │
│  └──────────────────────────────────────────────┘       │
│       │                                    │             │
│       ▼                                    ▼             │
│  ┌──────────────────────────────────────────────┐       │
│  │        Telephony Service (FastAPI)           │       │
│  │  - Call state management                     │       │
│  │  - WebSocket Media Streams                   │       │
│  │  - Mode routing (scripted/interactive)       │       │
│  └──────────────────────────────────────────────┘       │
│       │                                                  │
│       ▼                                                  │
│  ┌──────────────────────────────────────────────┐       │
│  │         Pipecat Voice Pipeline               │       │
│  │  Deepgram STT → GPT-4o-mini → ElevenLabs TTS│       │
│  └──────────────────────────────────────────────┘       │
│       │                                                  │
│       ▼ (when needed)                                    │
│  ┌──────────────────────────────────────────────┐       │
│  │         Norns Agent (LangGraph)              │       │
│  │  - Task creation, calendar, home control     │       │
│  │  - Full conversation history                 │       │
│  └──────────────────────────────────────────────┘       │
└─────────────────────────────────────────────────────────┘

---

Outbound Calling

Making Calls via Norns Tools

Norns has two tools for making phone calls:

1. make_phone_call

await make_phone_call(
    phone_number="555-123-4567",
    mode="interactive",  # or "scripted"
    message="Dinner is ready!",  # Required for scripted mode
    greeting="Hey! This is The Norns calling."  # Optional for interactive
)

Parameters:

• phone_number (str): Phone number in any format (E.164, 10-digit US, etc.)
• mode (str): "interactive" or "scripted"
• message (str, optional): Message to deliver (required for scripted mode)
• greeting (str, optional): Custom greeting for interactive calls

Examples via Slack:

• "Call John at 555-123-4567" → Interactive mode
• "Call mom and tell her dinner is ready" → Scripted mode with message
• "Call the plumber and ask about the estimate" → Interactive with objective

2. check_call_status

await check_call_status(call_sid="CA1234567890abcdef")

Parameters:

• call_sid (str): The call SID (full or first 8 characters)

Returns: Current call status (initiated, ringing, in-progress, completed, etc.)

Making Calls via API

curl -X POST https://telephony.ravenhelm.dev/api/call/outbound \
  -H "X-API-Key: $NORNS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "to_number": "+15551234567",
    "mode": "interactive",
    "user_id": "701973d2-57e4-4c84-a2ec-ded996dcf676",
    "greeting": "Hey! This is The Norns calling.",
    "objective": "Ask about their availability for the meeting tomorrow"
  }'

Request Body:

{
  "to_number": "string (E.164 format)",
  "mode": "scripted | interactive",
  "user_id": "string (UUID)",
  "message": "string (required for scripted)",
  "greeting": "string (optional)",
  "objective": "string (optional - goal of the call)",
  "system_prompt": "string (optional - custom prompt)",
  "callback_url": "string (optional - webhook for status updates)"
}

Response:

{
  "success": true,
  "call_sid": "CA1234567890abcdef",
  "status": "initiated"
}

---

Call Modes

Scripted Mode

Purpose: Deliver a specific message and hang up (one-way communication).

Flow:

1. Call connects
2. Norns delivers the message via TTS
3. Says "Goodbye!"
4. Hangs up

Example:

curl -X POST https://telephony.ravenhelm.dev/api/call/outbound \
  -H "X-API-Key: $NORNS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "to_number": "+15551234567",
    "mode": "scripted",
    "user_id": "701973d2-57e4-4c84-a2ec-ded996dcf676",
    "message": "This is a reminder that your appointment is tomorrow at 2 PM."
  }'

Use Cases:

• Appointment reminders
• Delivery notifications
• System alerts
• Simple announcements

Interactive Mode

Purpose: Full bidirectional conversation with objectives.

Flow:

1. Call connects
2. Norns delivers greeting
3. Conversation begins (voice LLM handles the dialogue)
4. When user requests a task/reminder/calendar event, delegates to Norns Agent
5. Norns Agent receives full conversation history for context
6. When complete, Norns says goodbye and hangs up

Example:

curl -X POST https://telephony.ravenhelm.dev/api/call/outbound \
  -H "X-API-Key: $NORNS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "to_number": "+15551234567",
    "mode": "interactive",
    "user_id": "701973d2-57e4-4c84-a2ec-ded996dcf676",
    "greeting": "Hey! This is The Norns. I wanted to follow up on the project.",
    "objective": "Ask about the status of the quarterly report"
  }'

Use Cases:

• Follow-up calls
• Information gathering
• Task creation via voice
• Complex multi-turn conversations

---

System Prompts

Inbound Calls (Default)

You are The Norns, a voice assistant. Always respond in English.
Keep responses SHORT (1-3 sentences).

You MUST use delegate_to_norns for ANY request that involves:
- Creating tasks, reminders, or calendar events
- Checking schedules or to-do lists
- Home automation commands
- Any action that requires memory or tools

For simple conversation (greetings, goodbyes, small talk), respond 
directly without delegation.

When the conversation naturally concludes, use end_call to hang up 
gracefully.

Outbound Interactive Calls

You are The Norns, making an outbound phone call. Always respond in English.
Keep responses SHORT (1-3 sentences). Be friendly and conversational.

YOUR OBJECTIVE: {objective}

IMPORTANT RULES:
1. Handle the conversation yourself - DO NOT delegate to Norns unless 
   the user asks to create a task, reminder, or calendar event.
2. Stay focused on your objective.
3. When the objective is complete OR the user wants to end the call:
   - Say a warm goodbye like "Thanks so much for your time! Have a 
     wonderful day!"
   - Use the end_call tool to hang up

Only use delegate_to_norns if the user explicitly asks for something like:
- "Add this to my calendar"
- "Remind me to..."
- "Create a task for..."

---

Call State Management

The telephony service maintains state for all active calls:

@dataclass
class CallState:
    call_sid: str
    direction: CallDirection  # INBOUND | OUTBOUND
    caller_phone: str
    called_phone: str
    user_id: str
    display_name: str = "Unknown"
    mode: CallMode = CallMode.INTERACTIVE
    outbound_config: Optional[OutboundCallConfig] = None
    status: str = "initiated"
    stream_sid: Optional[str] = None
    conversation_history: list = field(default_factory=list)
    llm_context: Optional[object] = None

State Lifecycle:

1. Initiated: Call created, Twilio dialing
2. Ringing: Phone is ringing
3. In-progress: Call connected, conversation active
4. Completed: Call ended normally
5. Failed/Busy/No-answer: Terminal error states

Cleanup: Call state is automatically cleaned up 60 seconds after terminal state.

---

Delegation to Norns

When Does Delegation Occur?

The voice LLM (GPT-4o-mini) delegates to Norns when the user requests:

• Task creation
• Calendar events
• Reminders
• Home automation commands
• Any operation requiring tools or memory

How Delegation Works

1. User makes request: "Add five trash bags to the shopping list"
2. Voice LLM invokes: delegate_to_norns(request="Add five trash bags to the shopping list")
3. Telephony service calls Norns API:

   POST /api/message
   {
       "message": "Add five trash bags to the shopping list",
       "user_id": "701973d2-57e4-4c84-a2ec-ded996dcf676",
       "conversation_history": [
           {"role": "user", "content": "Hey Norns"},
           {"role": "assistant", "content": "Hi! How can I help?"},
           {"role": "user", "content": "Add five trash bags to the shopping list"}
       ]
   }

4. Norns processes with full conversation context
5. Response returned to voice pipeline
6. TTS speaks Norns' response

Conversation History

Critical: The full conversation history is passed to Norns for context, including:

• For outbound calls: The objective is prepended as a system message
• All user and assistant messages from the live voice conversation
• This enables Norns to understand the full context when delegating

---

API Endpoints

Outbound Call Management

Endpoint	Method	Description
/api/call/outbound	POST	Initiate an outbound call
/api/call/{call_sid}/status	GET	Get call status
/api/calls/active	GET	List all active calls

Twilio Webhooks

Endpoint	Method	Description
/twilio/webhook	POST	Inbound call webhook (returns TwiML)
/twilio/outbound-webhook	POST	Outbound call connect webhook
/twilio/status-callback	POST	Call status updates
/twilio/stream	WebSocket	Media stream audio pipeline

---

Voice Pipeline

The Pipecat voice pipeline handles real-time audio processing:

Twilio Media Stream (WebSocket)
    │
    ▼
┌────────────────────────────────────────┐
│         TwilioFrameSerializer          │
└────────────────────────────────────────┘
    │
    ▼
┌────────────────────────────────────────┐
│   Deepgram STT (Speech-to-Text)        │
│   - Real-time transcription            │
│   - VAD (Voice Activity Detection)     │
└────────────────────────────────────────┘
    │
    ▼
┌────────────────────────────────────────┐
│   GPT-4o-mini (Lightweight LLM)        │
│   - Conversation routing               │
│   - Tool invocation (delegate/end)     │
│   - Response generation                │
└────────────────────────────────────────┘
    │
    ▼
┌────────────────────────────────────────┐
│   ElevenLabs TTS (Text-to-Speech)      │
│   - Natural voice synthesis            │
│   - Low latency streaming              │
└────────────────────────────────────────┘
    │
    ▼
Back to Twilio Media Stream

Tools Available to Voice LLM:

• delegate_to_norns - Delegate request to Norns Agent
• end_call - Gracefully end the call

---

Environment Variables

Variable	Description
WEBHOOK_HOST	Public hostname (telephony.ravenhelm.dev)
NORNS_URL	Norns API URL (http://docs/AI-ML-Platform/norns-agent:8000)
NORNS_API_KEY	API key for Norns authentication
TWILIO_ACCOUNT_SID	Twilio account SID
TWILIO_AUTH_TOKEN	Twilio auth token
TWILIO_PHONE_NUMBER	Norns phone number (+17372143330)
DEEPGRAM_API_KEY	Deepgram STT API key
OPENAI_API_KEY	OpenAI API key (for GPT-4o-mini)
ELEVENLABS_API_KEY	ElevenLabs TTS API key
ELEVENLABS_VOICE_ID	ElevenLabs voice ID

---

Deployment

Files

File	Purpose
/Users/ravenhelm/ravenhelm/services/telephony/main.py	FastAPI + Pipecat + Twilio integration
/Users/ravenhelm/ravenhelm/docs/AI-ML-Platform/norns-agent/agent/tools.py	Norns tools (make_phone_call, check_call_status)
/Users/ravenhelm/ravenhelm/docs/AI-ML-Platform/norns-agent/agent/graph.py	System prompt with call examples

Restart Telephony Service

ssh ravenhelm@100.115.101.81
cd /Users/ravenhelm/ravenhelm/services/telephony
docker compose restart

Restart Norns Agent

ssh ravenhelm@100.115.101.81
docker restart norns-agent

---

Usage Examples

Example 1: Simple Notification (Scripted)

Via Slack to Norns:

User: "Call Sarah at 555-1234 and tell her the package has arrived"

What happens:

1. Norns calls make_phone_call(phone_number="555-1234", mode="scripted", message="The package has arrived")
2. Telephony service initiates call via Twilio
3. When Sarah answers: "The package has arrived. Goodbye!"
4. Call ends

Example 2: Task Creation (Interactive)

Via API:

curl -X POST https://telephony.ravenhelm.dev/api/call/outbound \
  -H "X-API-Key: $NORNS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "to_number": "+15551234567",
    "mode": "interactive",
    "user_id": "701973d2-57e4-4c84-a2ec-ded996dcf676",
    "greeting": "Hey! Just calling to follow up on the shopping list.",
    "objective": "Get items for the grocery list"
  }'

Call flow:

Norns: "Hey! Just calling to follow up on the shopping list."
User: "Oh great! I need milk, eggs, and bread."
Norns: "Would you like me to add those to your list?"
User: "Yes please"
Norns: [delegates to Norns Agent with history]
       "I've added milk, eggs, and bread to your shopping list!"
User: "Thanks!"
Norns: "You're welcome! Have a great day!"
       [uses end_call tool]

Example 3: Status Check

Via Slack:

User: "What's the status of call CA12345678?"
Norns: [calls check_call_status("CA12345678")]
       "The call is currently in-progress."

---

Monitoring

View Logs

# Telephony service
docker logs -f telephony

# Norns agent (for tool invocations)
docker logs -f norns-agent

Check Active Calls

curl -H "X-API-Key: $NORNS_API_KEY" \
  https://telephony.ravenhelm.dev/api/calls/active

Check Health

curl https://telephony.ravenhelm.dev/health

---

Troubleshooting

Issue: Call Fails to Connect

Symptoms: Call status shows "failed" or "no-answer"

Diagnosis:

# Check call status
curl -H "X-API-Key: $NORNS_API_KEY" \
  https://telephony.ravenhelm.dev/api/call/{call_sid}/status

# View telephony logs
docker logs telephony | grep -A 10 "call_sid"

Solutions:

1. Verify phone number format (E.164 recommended)
2. Check Twilio account balance
3. Verify phone number is not blocked
4. Check Twilio console for errors

Issue: Voice LLM Not Delegating to Norns

Symptoms: Norns doesn't create tasks when requested during call

Diagnosis:

# Check telephony logs for delegation
docker logs telephony | grep "Delegating to Norns"

# Check Norns logs for incoming messages
docker logs norns-agent | grep "/api/message"

Solutions:

1. Verify NORNS_URL and NORNS_API_KEY are correct
2. Check that Norns Agent is running
3. Review conversation history in logs

Issue: Audio Quality Problems

Symptoms: Choppy audio, delays, echo

Diagnosis:

# Check for dropped frames
docker logs telephony | grep -i "error\|timeout"

Solutions:

1. Check network connectivity to Deepgram and ElevenLabs
2. Verify API keys for STT/TTS services
3. Review Twilio Media Streams diagnostics

Issue: Call Doesn't End

Symptoms: Call stays in "in-progress" after conversation completes

Diagnosis:

# Check for end_call tool invocation
docker logs telephony | grep "end_call"

Solutions:

1. Voice LLM may not be invoking end_call - review system prompt
2. Manually end via Twilio console if needed
3. State will auto-cleanup after 60 seconds in terminal state

---

Best Practices

For Scripted Calls

1. Keep messages concise - Under 30 seconds
2. Include call-to-action - "Reply to this email" or "Check your calendar"
3. Test messages - Ensure clarity and natural pacing
4. Provide context - "This is The Norns calling about..."

For Interactive Calls

1. Set clear objectives - Help the LLM stay focused
2. Use natural greetings - "Hey, it's Norns calling about the meeting"
3. Let the LLM handle conversation - Don't over-engineer the system prompt
4. Trust delegation - When user wants tasks/events, the LLM will delegate
5. Monitor conversation history - Ensure full context is passed to Norns

API Integration

1. Always use E.164 format - "+1XXXXXXXXXX"
2. Set callback URLs - Track call completion asynchronously
3. Store call_sid - For status checks and debugging
4. Handle errors gracefully - Network, API, and user errors

---

Related Documentation

• Telephony Service - Inbound calling and voice gateway
• Norns Agent - Core agent capabilities
• Norns Memory System - Long-term memory architecture
• Voice Platform - LiveKit, Piper, Whisper integration

---

Recent Updates

2026-01-04: Outbound Calling Feature

• Added outbound call support with scripted and interactive modes
• Implemented make_phone_call and check_call_status tools in Norns
• Added call state management and conversation history tracking
• Integrated objective-based system prompts for focused outbound calls
• Full delegation to Norns with conversation context

Modified Files:

• /Users/ravenhelm/ravenhelm/services/telephony/main.py
• /Users/ravenhelm/ravenhelm/docs/AI-ML-Platform/norns-agent/agent/tools.py
• /Users/ravenhelm/ravenhelm/docs/AI-ML-Platform/norns-agent/agent/graph.py

2026-01-07: Enhanced Delegation Support

Updated outbound call system prompt to allow broader delegation types:

New delegation triggers:

• "Send me that on Slack" → Delegates to Norns for Slack messaging
• "Message me the directions" → Delegates for Slack delivery
• "Look that up and send it" → Delegates for info lookup + messaging
• Any request requiring action outside the phone call

Updated System Prompt:

Use delegate_to_norns when the user asks you to:
- Send them something on Slack/message them
- Add something to their calendar
- Create a reminder or task
- Look something up (directions, info, etc.) and send it to them
- Any request that requires action outside of this phone call

When delegating, clearly describe what the user wants in the request parameter.

Modified Files:

• /Users/ravenhelm/ravenhelm/services/telephony/main.py (OUTBOUND_SYSTEM_PROMPT)
• /Users/ravenhelm/ravenhelm/docs/AI-ML-Platform/norns-agent/agent/agents/workers/communication_workers.py (PhoneCallWorker auth)