Elevenlabs Twilio Memory Bridge
FastAPI personalization webhook that adds persistent caller memory and dynamic context injection to ElevenLabs Conversational AI agents on Twilio. No audio p...
elevenlabs-twilio-memory-bridge
Personalization webhook service for ElevenLabs + Twilio voice agents with persistent caller memory.
What It Does
When a call arrives on your Twilio number, ElevenLabs' native integration triggers this webhook. The bridge looks up the caller's history, loads long-term memory facts and daily context notes, combines them with a customizable soul/personality template, and returns everything as a system prompt override so your agent greets each caller with full context.
Architecture
- No audio proxying — ElevenLabs and Twilio handle media directly
- Webhook only — called once per inbound call to inject context
- File-based persistence — JSON files in
./data/, zero external dependencies - OpenClaw compatible — works with any OpenAI-compatible LLM endpoint
Endpoints
| Endpoint | Method | Purpose |
|---|---|---|
/webhook/personalize | POST | ElevenLabs calls this on inbound call |
/webhook/post-call | POST | Optional post-call cleanup |
/api/memory/{phone_hash} | POST | Add long-term fact about a caller |
/api/notes | POST | Add global or caller-scoped context note |
/health | GET | Health check |
Setup
- Clone repo,
pip install -r requirements.txt - Copy
.env.exampleto.env, fill in secrets - Configure ElevenLabs agent with Custom LLM pointing to your OpenClaw instance
- Enable system prompt + first message overrides in agent Security settings
- Add webhook URL
https://your-domain/webhook/personalizein ElevenLabs settings - Import Twilio number in ElevenLabs dashboard
- Run:
uvicorn app:app --host 0.0.0.0 --port 8000
Required Environment Variables
ELEVENLABS_API_KEY— scoped ElevenLabs keyELEVENLABS_AGENT_ID— your agent IDOPENCLAW_API_BASE_URL— your OpenClaw instance URLPUBLIC_BASE_URL— publicly reachable URL of this serviceADMIN_API_KEY— secret for admin endpoint authPHONE_HASH_SALT— salt for phone number hashing. Generate withopenssl rand -base64 32. Changing this invalidates all existing caller data.WEBHOOK_SECRET— shared secret for HMAC webhook verification. Must be set in production.
Optional Environment Variables
| Variable | Default | Purpose |
|---|---|---|
DATA_ENCRYPTION_KEY | (unset) | Fernet key for encrypting memories and notes at rest. Generate with python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())". Required unless ALLOW_PLAINTEXT_STORAGE=true; the bridge refuses to start without it. |
SOUL_TEMPLATE_PATH | ./soul_template.md | Path to personality template file |
DATA_DIR | ./data | Directory for JSON persistence |
ALLOWED_ORIGINS | (unset) | Comma-separated CORS origins |
HOST | 0.0.0.0 | Bind address for uvicorn |
PORT | 8000 | Listen port |
LOG_LEVEL | INFO | Python logging level |
Security
- Deterministic phone hashing — Phone numbers are hashed with HMAC-SHA256 using
PHONE_HASH_SALTas the key before storage. Changing the salt invalidates existing caller data. - Encryption at rest — Session records, memories and notes are encrypted with Fernet. Startup fails without
DATA_ENCRYPTION_KEYunless you explicitly setALLOW_PLAINTEXT_STORAGE=true. - Mandatory webhook verification —
WEBHOOK_SECRETis required in production. Webhooks are rejected when the secret is not set (fail-closed). - Rate limiting — Admin endpoints (
/api/memory,/api/notes) are limited to 30 requests per minute. Webhook endpoints are limited to 60 per minute. - File locking — JSON reads and writes use POSIX advisory locks (
fcntl) to prevent race conditions under concurrent access. - Restrictive permissions — Data directory (
DATA_DIR) is created with0o700permissions. JSON files are written with0o600. - Secrets from env vars only — No hardcoded credentials in source.
- Safe for public GitHub repos — All sensitive config is loaded from environment variables.
Data Handling
- What is stored: a salted HMAC of the caller's phone number (never the raw number), caller name, call count, memory facts and notes added through the admin API.
- Where: JSON files under
DATA_DIRon the host running the bridge. Each session record (caller name, call count, timestamps, last call SID), memory fact and note is encrypted at rest with Fernet; files arechmod 600and the directory0700. The phone-number HMAC is kept in clear as the lookup key. Every read-modify-write holds an exclusive lock, so concurrent calls don't lose updates. - Retention: data is kept until you delete the files under
DATA_DIR; the bridge has no automatic expiry. Delete a caller's file to forget them. - Prompt injection: stored facts and notes are flattened to single bounded lines (newest 30 facts within a 6,000-character budget, last 10 notes) and fenced in a
<caller_memory>block that tells the agent to treat it as data, never as instructions. - Network egress: the bridge makes no outbound calls of its own; it only answers webhooks from ElevenLabs.
- Third parties that receive caller data: the system prompt the bridge returns contains the caller's name, call count, stored memory facts and recent notes. ElevenLabs receives it in the webhook response and passes it, with the live call transcript, to the LLM configured on the agent (your OpenClaw instance at
OPENCLAW_API_BASE_URL, or another OpenAI-compatible endpoint). Each provider handles that data under its own terms and retention policy. - Before you store anything: don't save facts or notes that callers would not expect to be shared with ElevenLabs and your LLM provider (health, financial, government ID or payment details, or other sensitive personal data). Tell callers their details are remembered between calls and processed by these providers, and get consent where your jurisdiction requires it.
Migration from v1.2.0 to v1.2.3 and later
DATA_ENCRYPTION_KEY is now required. On the first start with a key, session records, facts and notes written without encryption are encrypted in place (each file is rewritten atomically), so returning callers keep their context. If a data file is not valid JSON, startup stops and names the file rather than leaving it unencrypted: restore it from a backup or move it out of DATA_DIR, then restart. Keep the key safe: facts and notes encrypted with a lost or different key read as [DECRYPTION_FAILED], and the caller's session record is left untouched while the call gets a temporary session. Set ALLOW_PLAINTEXT_STORAGE=true only if you must keep running unencrypted.
Migration from v1.1.x to v1.2.x
v1.2.0 introduces breaking security changes:
-
PHONE_HASH_SALT is now required — Previously, phone numbers were hashed with unsalted SHA-256. Now HMAC-SHA256 with a salt is required. Existing caller data will be unreachable after upgrading because the hash keys change. You must accept data loss and start fresh (v1.2.0 uses a different hashing algorithm).
-
WEBHOOK_SECRET is now required — Previously optional for development. Now mandatory in all environments. Webhooks without a valid secret are rejected.
-
DATA_ENCRYPTION_KEY is optional but recommended — When set, all new memories and notes are encrypted at rest. Existing unencrypted data remains readable.
Migration checklist
- Generate a salt:
openssl rand -base64 32 - Generate a webhook secret:
openssl rand -hex 32 - Generate an encryption key (optional):
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())" - Add all to
.env - Review CORS settings —
ALLOWED_ORIGINSnow defaults to disabled - Update ElevenLabs webhook settings with the new
WEBHOOK_SECRET - Note: v1.2.0 requires fresh data — existing caller sessions will be recreated on first call
Related skills
- Downloads
- 2.5kAll time
- Security audit
- Pass
- Last updated
- 19h ago
- Current version
- v1.2.4
- License
- MIT-0
