for gpn24 i built a self-hosted voice assistant you could call from any dect phone on the conference network. dial 7244, wait through some jazz if the line is busy, and a local llm picks up, greets you by name, and tells you which talks are starting soon.

i had 3 days, 0 euros, and a pile of mismatched gpus, two of them a decade old. what followed was a lot of cursed cuda debugging, some genuinely unresolved latency problems, and (somewhat to my own surprise) a hotline that actually worked and pointed me to one of the highlights of my event.

the architecture, at a glance:

caller (dect client)
  ^
  └> asterisk (sip; relay via bidirectional audiosocket)
      ^
      └> orchestrator/harness
         (vad -> stt -> llm(tools: schedule info, memory, etc) -> tts)

if you want to hear a (very high latency) result before committing to the read, skip to the demo.

why a hotline?

isnt that tech of the past? who even calls these days, duh?

39c3 was my very first real chaos event. it was a lot, a lot of lights, interesting people, talks, interesting things people hacked on and quite little sleep. i really enjoyed it!

among the things i had not yet experienced before was the dect network that gets put up at chaos events, this means that everyone with a dect client (meaning a basic old land line wireless handheld phone) could register in the network and get a 4-6 digit phone number, this could then be used to reach any other person with a dect client on the premises of the event. i found it to be a really nice communication style as it was on the one hand a really funny and nerdy addition to the event while on the other hand it increased the immersion the event provided as it was not necessary to look at ones phone all the time.

after 39c3 the idea of setting up my own dect hotline emerged in my mind. i definitely would not be the first one, dect hotlines had been the source of many funny moments my friends and i had shared at the event.

these hotlines could generally be classified into two classes:

• non-interactive: these usually play a funny audio, sometimes always the same one, sometimes randomly rotating/generating them. examples of this can be:
  • song based hotlines (for example a hotline that plays the star wars cantina band theme in a loop)
  • the “get your personal private key” hotline, which would on call generate a public/private ssh keypair and reads you your private key out loud via the phone line.
  • the “departing trains” hotline which would read out loud all the departing trains that are leaving in the next minutes from the train station next door (actually really handy).
• interactive: these can be anything, honestly most of them provide the illusion of interactivity, examples of this i saw were:
  • the horoscope hotline, here you could provide your star sign by typing a number between 1-5. the star sign options you could choose from were along the lines of “sweaty sock”, “tired cat”, “active potato”, after you chose you would get a random reading of your future.
  • the dating hotline where you were put in a waiting queue until a second person called in at the dating hotline after which the two of you would be matched and could have a chat.

❮

❯

after the conference i was a bit hooked and on the train back home from hamburg already started thinking about possible fun concepts for a hotline. i wanted something that was technically challenging and either useful or funny.

a few days went by after the conference, new years happened and i forgot about it, or better said, i put it on hold for now. some months later i was able to buy tickets for gpn24, another chaos event. and so it came that i had planned to go to karlsruhe on the 4th of june. just a few days before that i remembered that i still wanted to host a dect hotline so i started brainstorming on what i wanted to do.

in the few weeks before the event i had been interested in local agentic workflows. as i didnt have the budget and also was handling sensitive information i did not want to share it with the big llm apis. this made me dive deeper into selfhosted llms, building agentic workflows, harness building etc. as i was into that topic anyway i thought why not have some fun and implement a dect voice assistant hotline for the event. i was not sure about the exact capabilities but i thought it would be a fun exercise and i could adjust the harness as the event went on. so i started.

constraint & setup

to spell it out, this was:

• 3 days
• 1 person
• 0 euro

it all started on the 1st of june. before i could start building my agentic harness i first needed the components that it would all be built on.

we can generally assume that the architecture can be split into two parts: the telephony audio and control side (handling calls, phone button presses, audio in&out streaming, …) and the processing of the audio (an llm agent in my case).

hardware

before getting into any of the software i need to get the essential compute out of the way, especially gpu compute as that is sadly necessary for local realtime model inference.

3 days before the event i scraped together everything i could find that quickly, which meant i had one gpu of my own and borrowed 2 older gpus as they were the only things i was able to come by. its a bit of a mess to be totally honest. it looks like this:

┌──────────────────── same local network ──────────────────────┐
│  ┌────────────────────────┐      ┌────────────────────────┐  │
│  │      homelab:g0        │      │       gpu-system       │  │
│  │                        │      │                        │  │
│  │ cpu: intel i7-13700    │      │ cpu: intel i7-6700     │  │
│  │ gpu: rtx 3090          │      │ gpu 0: gtx 1080        │  │
│  │ vram: 24 gb            │      │ vram: 8 gb             │  │
│  │                        │      │                        │  │
│  │                        │      │ gpu 1: gtx 1080        │  │
│  │                        │      │ vram: 8 gb             │  │
│  └────────────────────────┘      └────────────────────────┘  │
└──────────────────────────────────────────────────────────────┘
 disclaimer: llm generated base ascii, further adapted by me

while the rtx 3090 is my own (read more about it at my homelab blogpost), the gpu-system is sourced externally for the duration of the gpn weekend, it is only borrowed, i dont own those gpus. as you can also see that system is very dated (it was top of the line almost exactly 10 years ago). while my entire homelab is on nixos i quickly put a headless debian on the gpu-system and went on. this is not anything permanent but simply a fun project for the weekend.

the additional gpu-system was necessary as hosting 3 models (more on that in the next section) inside of 24gbs of ram (with high enough quality models) is not possible and loading and unloading each model as necessary (which is technically possible) would have come with very large latency penalties.

now i had 40gb vram to work with and the project could start (sadly the gtx 1080s might have been a very inoptimal choice but more on that later). 2 more days to go :)

telephony

another basis for the project that needed to be established before looking into anything agentic was the telephony architecture.

although the phone network at chaos events is commonly associated with dect, eventphone also provides voip access through sip. dect is well suited for portable handsets, whereas sip is a much better fit for connecting a hotline server.

sip itself mostly handles call signaling: establishing, routing and ending the call. the actual audio is transported separately, usually through rtp.

i did not want the agent application to implement any of this directly. telephony servers are very mature software and already solve most of the boring but surprisingly complicated parts of operating a phone service. commercial options include amazon connect (aws) and twilio programmable voice, but for this project i chose the free and open-source asterisk project.

asterisk connects to the eventphone sip trunk and takes care of the complete phone-facing side of the system: accepting calls, playing an initial consent announcement, routing callers, placing them into a waiting queue and playing music on hold (🥰).

once the single agent slot becomes available (more on this limitation later), asterisk forwards the call to the orchestrator using its audiosocket() application. this provides a direct bidirectional raw-pcm tcp connection: asterisk continues to deal with sip and rtp, while the agent only has to receive and return audio samples.

this separation was quite useful. callers waiting in the queue never consume model capacity, and only one call reaches the agent pipeline at a time. the orchestrator also rejects additional audiosocket connections as a final safeguard.

the whole asterisk setup runs in docker, with the sip credentials kept outside the repository and the audiosocket interface bound only to localhost. for a temporary conference hotline this gave me most of the features of a proper call center without having to make the agent code understand telephony protocols.

this is also how the project name sip-slop came to be. it is (llm-)slop delivered via a sip connection. (i deliberately dont use the word ai btw, llms are not ai in my opinion, ai is a word without a truly defined meaning and should only be used with a wink of an eye).

the voice pipeline

with the basis set it was time to think about how to best handle the incoming voice data.

architecture

finally we can start with the first agent related question: what model architecture do i even want?

to my knowledge there are two ways of building a voice agent:

1. an end-to-end (e2e) speech model (for example Qwen3-omni) that natively takes in audio, reasons on it and natively produces audio again. one integrated model doing everything.
2. the classic stt -> llm -> tts pipeline: a normal text llm with a speech-to-text model in front and a text-to-speech model behind it.

the e2e approach is tempting. fewer model boundaries mean less application-level plumbing and potentially lower latency, since there are no intermediate data-transfer steps between models. and because the audio never gets flattened into text, acoustic information like tone, emphasis and hesitation survives all the way into the model instead of being lost or reduced to textual descriptions. turn-taking and speech generation also tend to feel more natural.

the price you pay is control. the model is a black box between “sound in” and “sound out”: the application gets little access to intermediate representations, which makes the system hard to observe and even harder to validate. by the time you could check what the model is saying, the audio is already streaming into someones ear. orchestration around e2e models is also generally less mature and more vendor-specific, particularly for tool calling, rag, structured outputs and agentic workflows. and since its all one model, you cant swap out or optimize individual parts.

the pipeline approach inverts most of this. everything in the middle is text, so transcripts, llm history, tool calls and the exact text handed to tts are all trivially loggable. text is where all the mature tooling lives: monitoring, redaction, safety classifiers, approval steps, plus the entire existing ecosystem of agent loops, rag and structured outputs. each component can be chosen independently for its own hardware, latency, cost, language and quality constraints, with far less vendor lock-in, and there are simply more models to choose from at every stage. when something breaks, you can usually tell which stage broke.

the downsides are real though. more serial stages means more latency (foreshadowing!), although streaming and pipeline overlap can compensate for a lot of it. acoustic nuance dies at the stt boundary, and stt errors dont just disappear, they propagate into the llm and shape the final response. and you now own the orchestration complexity of partial transcripts, streaming responses, tts buffering, interruptions and cancellation across three independent models.

in case you are interested in mid-generation safety analysis, see my blogpost about my mats submission to neel nanda on activation space safety classification (available soon™)

for this project the decision was pretty clear. i went with stt -> llm -> tts because:

• i dont have money, i have a weird pile of mismatched hardware. three separate models can be split across gpus however they fit, a single big e2e model cant.
• tools are the whole point of the hotline (schedule lookup, memory, …), and tool calling with a text llm is a solved problem. with e2e models it mostly isnt.
• i already know my local text llms. despite what somebody on the internet claims every week, local models have not made the big api vendors obsolete. finding one that runs well, fits in vram with enough kv-cache, and follows instructions instead of rambling into endless loops takes real evaluation work. i had already done that work for text llms. for local e2e speech models i would have been starting from zero, three days before the event.

picking models

finally i was at the stage of selecting the actual models i wanted to use. this was not as simple as it seemed.

there are three models we have to choose here as talked about before. this process is sadly not trivial as its not about simply choosing the best possible model but a consideration of model quality, latency/generation time, vram size, context (kv-cache) etc.

llm

in the past i already had been experimenting a lot with local models and had also benchmarked a number of models. for anyone interested i really recommend the youtube channel protorikis as that is the only youtube channel i know of that actually talks about the relevant metrics when it comes to local llms. many reviews instead focus almost entirely on benchmark scores and substantially overstate how close local models are to frontier proprietary systems

my 4 best options (as of my awareness, at the time of implementing) were gemma4-31b, gemma4-26b-a4b, qwen3.6-35b-a3b, qwen3.6-27b. basically this decision can be summarized as gemma4 vs qwen3.6 and dense vs moe.

after some testing on summary and tool calling tasks i liked the results i got from qwen3.6 a lot more, while gemma4 felt like the usual tiny llms qwen3.6 for the first time felt somewhat useable. it followed instructions, didnt get into endless loops, wasnt hallucinating too bad, was able to perform some logic tasks and was actually able to use tools. somehow qwen3.6 felt better, its sad that the current llm benchmark landscape is so insufficient, that i have to go off vibes as benchmarks numbers dont really tell a faithful image of a models capability.

next was moe vs dense. this distinction is often described online as “getting the intelligence of a large model at the speed of a small one,” which is a bit misleading.

a mixture-of-experts model stores many parameters but only activates a subset of its experts for each token. this reduces the amount of computation required during inference, although the complete model still has to fit into memory. a 35b-a3b model therefore occupies roughly the memory of a 35b model while using around 3b active parameters per token, plus the work performed by shared components such as the attention layers.

this does not make it equivalent to either a dense 3b or dense 35b model. the additional experts increase its capacity, but total parameter count alone does not tell you how capable the model will be.

for local inference, my simplified version of the trade-off was:

• dense models were the safer choice when i wanted to maximize capability within a fixed amount of vram.
• moe models were attractive when the complete model fit into memory and higher token throughput mattered.

these are heuristics rather than universal rules. architecture, training data and post-training can easily matter more than whether a model is dense or sparse.

for previous projects involving logically demanding, long-running background tasks, i had used the dense qwen3.6-27b. latency mattered relatively little there, so i could prioritize capability over generation speed.

for sip-slop the priorities were reversed. it was possible to improve the 27b models performance using speculative decoding, reaching around 38 tokens per second normally and 61 with mtp on my rtx 3090. qwen3.6-35b-a3b, however, reached around 140 tokens per second without relying on an mtp setup (currently there is no qwen3.6-35b-a3b mtp implementation which improves the models performance). (see 27b benchmark, 35b-a3b benchmark)

the reduced capability was completely acceptable here. a hotline does not require especially deep reasoning, while slow responses immediately break the conversational illusion. and if the model occasionally said something stupid, that felt more like a comedic feature than a serious service degradation.

i see the “large-model intelligence at small-model speed” description quite often, so an earlier version of this section turned into a small rant. for a more complete treatment, the unified scaling laws for routed language models paper discusses the relationship between model size and compute, while reiner popes appearance on the dwarkesh podcast provides a more conversational explanation of the training and serving dynamics.

lastly, there was quantization. an rtx 3090 only has 24gb of vram, while 35b parameters stored in bf16 require ~71gb for the weights alone. trivially, that does not fit.

i used unsloths qwen3.6-35b-a3b-ud-iq4_xs.gguf, an approximately 4-bit gguf quantization weighing 18.2gb. together with a quantized kv cache, this allowed me to configure a context size of 163k tokens.

that context length was almost certainly overkill, but i wanted some flexibility for longer phone conversations, tool outputs and whatever increasingly elaborate agent harness i might build later. in practice, filling anything close to the complete context would likely have introduced substantial context degradation (aka context rot) long before running out of space.

i hosted the model locally through llama.cpp, which had the most mature gguf support for this setup, disabled thinking to reduce latency, and with that the llm side of the system was done.

tts

i ended up using cosyvoice3-0.5b-2512. other models i considered, and likely would have chosen under different circumstances, were qwen3-tts, pocket, fish s2 and chatterbox. but how did i get here?

the things that mattered most to me were voice quality, latency, vram usage, the ability to influence tone and compatibility with nvidias pascal architecture. for both tts and stt i wanted to stay below an rtf of 0.8, meaning inference should take less than 80% of the duration of the audio.

fish s2 sounds really good, has low latency and allows control over its delivery using inline tags such as [whisper] or [excited]. it therefore looked like a very good fit for the hotline.

then i tried to run it on a gtx 1080. it was almost comical how many issues i ran into.

pascal predates tensor cores and has no native bf16 support. while the gtx 1080 technically supports fp16, its fp16 throughput is extremely poor and the inference paths i tested were also numerically unstable. i encountered inf and nan probability tensors, device-side asserts and one prebuilt cuda wheel that simply died with a sigill.

for most pytorch implementations this effectively left me with fp32. with fp32 that meant 8gb (the memory size of my gtx 1080) would be filled by a 2b param model already, before caches, activations and the rest of the runtime are considered. this made my options quite slim.

quantization itself is not impossible on pascal, but it requires a backend with compatible kernels. i eventually found an unofficial gguf conversion of cosyvoice3 together with cosyvoice.cpp, a ggml-based c++ runtime. after compiling the cuda backend for compute capability 6.1, cosyvoice3-2512_q8_0.gguf finally ran reliably on the gtx 1080.

on a representative test it generated 4.4 seconds of audio in 2.33 seconds, resulting in an rtf of roughly 0.53. this was comfortably below real time, although the backend still generated the complete wav before playback could begin.

so i ended up with cosyvoice because it was the best-sounding model i could make fit into 8gb of vram, remain stable on pascal and stay below my latency target. most importantly, it actually worked.

stt

for speech-to-text i used parakeet tdt 0.6b v3, running through nvidias nemo framework on the other gtx 1080.

the criteria here were simpler. i needed reasonably accurate transcription in a variety of mostly european languages, low latency and a model small enough to fit comfortably on the remaining card.

once again, pascal complicated things. i loaded the model in fp32, which at roughly 600 million parameters still fit easily into 8gb of vram. getting a pytorch build that actually worked on compute capability 6.1 took some experimentation, so instead of trusting torch.cuda.get_arch_list() i validated the setup using a real fp32 numerical forward pass on both gpus.

the service buffers one complete utterance and only starts transcription once the caller has stopped speaking. technically, this is not streaming stt.

normally that would be a significant limitation, but parakeet turned out to be extremely fast. on an 11.57-second recording captured through the actual dect audio path, it achieved an rtf of approximately 0.015, meaning transcription took around 170 milliseconds.

at that speed, implementing streaming for the mvp would have saved very little. even a 30-second utterance would only take roughly half a second to transcribe after the caller stopped speaking. compared with the llm and tts stages, stt was already the fastest part of the pipeline.

so i kept the simpler buffered implementation. true streaming could still be useful later for partial transcripts or speculative processing, but for the first version it would mostly have added complexity to a component that was already fast enough.

getting all models operational took me an entire day. luckily i already had my llm model confed via my nixconf before this project so i only needed to set up stt and tts on pascal. with that done i had one day left to the event and my agent harness still needed to be implemented.

harness

with all three model endpoints running, i had one day left before the event. unfortunately, connecting three models in a row does not yet make a usable voice agent. i still needed the harness that would manage the conversation, call tools, remember callers and convince several independent components to behave like one system.

requirements & tools

before implementing it, i wrote down what i actually wanted the hotline to do.

at the most basic level, it should:

• greet callers when they connected.
• detect when somebody started and stopped speaking.
• let callers interrupt the agent while it was talking.
• start answering before the entire response had been generated.
• cancel outdated llm and tts work after an interruption.
• end the call cleanly when either side requested it.

the hotline should also be more useful than simply exposing a local chatbot over the phone. its main tool was therefore access to the gpn schedule. callers should be able to ask what was starting soon, where an event was taking place or request more information about a previously mentioned talk.

i imported the official fahrplan into a local database and exposed tools for searching it and retrieving individual events. smaller tools provided the current time, changed the voice or speaking style used by tts and allowed the agent to end the call after finishing its goodbye.

lastly, i wanted the agent to remember returning callers. not in the sense of building a serious personal profile, but enough to remember harmless details such as somebodys interests, music taste or a running joke from an earlier call.

eventphones phonebook already contained the nick each person had provided when registering. asterisk passed the caller number to the harness, allowing it to look up that nick and add it to the context automatically. this gave the agent a surprisingly effective opening trick: it could greet many callers by name before they had said anything.

the prompt still treated this identity as unverified. phone numbers can be spoofed, and somebody else might simply be holding the handset.

transcripts were stored in a local database after each call, but complete previous conversations were never inserted into later prompts. instead, a background task distilled them into deliberately harmless memories. only this sanitized summary was shown to the agent during future calls.

with the expected behaviour defined, i could start implementing the actual conversation loop.

conversation loop

incoming audio first passes through a small energy-based voice activity detector. because the hotline was intended for a loud event hall, i tried to make it estimate the ambient noise floor instead of relying on one fixed volume threshold.

when the caller begins speaking, any current llm generation and unfinished tts work are cancelled, and queued playback is discarded. this allows the caller to interrupt the agent, commonly called barge-in. even the initial greeting is interruptible, so callers do not have to politely wait for the machine to finish introducing itself.

once the caller stops speaking, the utterance enters the familiar pipeline: stt -> llm -> tts

stt processes the complete utterance at once. the llm response is streamed token by token. the harness groups the incoming text into sentence-sized chunks and submits each completed chunk to tts. while the first sentence is being spoken, the llm can continue generating the next one.

the tts backend itself is still buffered. it generates a complete wav for each submitted text chunk before returning any audio. the pipeline therefore overlaps later llm and tts work, but the route to the first spoken sentence remains completely serial.

tool calls are handled inside the same llm loop. the model can search the fahrplan, retrieve an individual event, change its voice, get the current time or request that the call end.

the fahrplan search tool contains some ordinary deterministic logic for time filters, relevance scoring and german umlaut normalization. this was far more reliable and efficient than placing the entire event schedule into the prompt and asking a small local model to please figure it out.

ending the call also had to be a tool. disconnecting immediately when the model requested it would have cut off its own goodbye, which would have been efficient but not especially polite. the harness therefore waits until the final audio has finished playing before hanging up.

all conversations, tool calls and derived memories are stored locally in sqlite. memory refactoring only runs while the hotline is idle and is cancelled immediately when a real caller arrives. reorganizing an old conversation while somebody waits on the phone seemed like the wrong scheduling priority although it would have further enforced my little assistants worker rights.

one caller at a time

another limitation was concurrency. my hardware could support exactly one active caller.

the llm context was configured much larger than necessary (32k would probably have been more than enough) but tts was the actual hard limit. one caller already kept the gtx 1080 busy enough that adding a second would have pushed audio generation below real time.

i therefore used asterisks waiting queue. if somebody wanted to talk to my virtual assistant while it was busy, they had to wait until the current call ended.

while waiting, they were played a jazz cover of a song by a frankfurt local rapper, which was my subtle attempt at spreading a bit of my citys culture.

implementation notes

with a fair bit of dedication and a good amount of llm assistance, i implemented a shaky but functional version in roughly half a day, mostly during the train ride from frankfurt to karlsruhe.

i wanted it finished before arriving. i knew that once i was at gpn, there would be too many people, talks and unrelated projects to suddenly become interested in carefully debugging my hotline.

i set up the asterisk right before starting the agent implementation. i did not use an agent framework. the complete conversation loop was self-made and connected directly to asterisk through audiosocket.

something such as langchain would not have helped much because most of the complexity came from audio transport, turn detection, cancellation and playback rather than from chaining text prompts together.

i did consider pipecat and livekit agents. both already provide many of the things i was building myself: streaming speech adapters, interruption handling, frame-based pipelines and telephony integrations.

the problem was that their easiest paths were built around common hosted model providers. adapting my collection of self-hosted and somewhat jinxed model endpoints would itself have required a fair amount of work. with only half a day available, hand-rolling the exact parts i needed felt simpler.

overall, i am still quite happy that i built it myself. i learned much more about the actual mechanics of a voice agent than i would have by connecting three provider apis.

that does not mean the result was polished.

the interruption detection worked at home but was unreliable on the very noisy gpn floor. longer conversations also exposed some odd model behaviour. after talking for several minutes, the llm would sometimes stop acting as if it knew that end_call existed, despite the tool still being present in its instructions (maybe operating a phone reliably will need agi).

as you can imagine the code is not clean at all, a lot of hot fixes and life lines, i therefore decided not to share it.

what went wrong

well, of course this project had its speed bumps, but with the timeline that was almost expected. in my opinion the most interesting part of projects is observing in what ways they failed and learning from that, that is why doing (no matter how bad) is always better than not doing things :)) (in a non critical context of course lol).

latency

the largest issue was latency.

despite the good standalone benchmarks, each conversational turn took somewhere between roughly five and fifteen seconds before the agent began speaking. tool use generally pushed it towards the slower end.

this made the hotline usable, but only for callers with patience.

i did not have time to investigate it properly during the event. afterwards, i inspected the implementation and found several likely contributors:

• vad hangover: the harness waits for around 700 milliseconds of silence before deciding that the caller has finished. this was intended to avoid cutting people off, but it adds almost a second before any later stage can begin. a better implementation could start speculative work during this period and cancel it if the caller resumes speaking.
• batch stt: transcription only begins after the complete utterance has ended. with its tiny rtf (~1%) this was probably not a major bottleneck, but partial transcription could still allow later stages to prepare sooner.
• llm prefix caching: a seconds-resolution timestamp was included in the system prompt and changed on every turn. this likely prevented llama.cpp from reusing its prompt-prefix cache, forcing it to process the system prompt, tool definitions and conversation history again each time.
• tool rounds: schedule questions can require several llm requests before any speakable text is produced. each additional tool or summarization round adds another prompt-processing delay.
• buffered tts: the first response chunk only reaches tts after the chunker has collected enough text, and cosyvoice then generates that complete chunk before returning the first audio byte. on the gtx 1080, this alone added another two to three seconds.

once playback started, the system behaved much better. later llm output and tts chunks could overlap with audio that was already playing. the main problem was therefore not sustained throughput but time to first audio.

the individual model rtf measurements suggest that the pascal gpus were not solely responsible. the delay was more likely the result of several serial stages, one probable cache-invalidation bug and a harness assembled on a train in half a day.

for a future event, i would first add proper per-stage timing instrumentation. after that, the likely improvements would be to keep the system-prompt prefix stable, begin speculative processing during the vad hangover, emit a much shorter first text chunk and use genuinely streaming tts output if the backend supports it.

while implementing i quickly hand rolled a vad, now that i looked deeper into it i found that there are mature open source options (for example silero-vad) which i would most definitely use instead next time.

all of these changes look manageable with a bit of patience. at gpn, however, i was busy with other projects, people i had met and the event itself. honestly, i was already quite happy that the hotline i had built in roughly two days answered the phone at all.

privacy, data protection & security

the chaos community is very privacy focused (and so am i), so the goal was simple: be transparent about what gets stored, and design so that the worst case is boring.

three things were stored, all in a local sqlite db on my own hardware: transcripts, tool call logs, and per-caller memories. the memories were not raw transcripts, a background task distilled each call into a few deliberately harmless notes (“likes jazz”, “asked about the nixos meetup”), and only those ever made it into future prompts. memories were keyed strictly to the caller number, so caller A’s notes could never show up in caller B’s context. nothing left my network, and after the event i deleted the whole db. there are no records anymore.

every flyer said transcripts are stored, and asterisk told you the same before connecting you. hang up during the announcement and nothing was stored.

now to the fun part, the threat model:

• caller-id spoofing. eventphone numbers are not identity, and the prompt treated the looked-up nick as unverified. spoof someone’s number and the maximum payout is their distilled memory — a few sanitized lines. that’s exactly why the distillation step exists: a successful impersonation learns your favorite music, not your conversation history.

• prompt injection. probably works! and mostly doesn’t matter. the isolation between callers is structural, not prompt-based. the agent’s context simply never contains anyone else’s data, and no amount of “ignore previous instructions” can leak what isn’t there. injection could at most make the agent misuse its own tools, which were fahrplan search, current time, voice change and end_call. all read-only or harmless. worst realistic jailbreak: the agent says something stupid in a funny voice. that was already within normal operating parameters.

• infrastructure. audiosocket bound to localhost only, sip credentials outside the repo, asterisk as the single thing facing the trunk. the agent process had no shell, no network tools. nothing beyond the sqlite db and the model endpoints.

lastly, the version i didn’t build… the original idea was the opposite: hook a full agent framework (hermes, turnstone) up to sip, give callers arbitrary code execution and make it a hacking playground that resets every k hours. i decided against it, because i did not trust myself to build a real sandbox in half a day. this was my own infra, not a throwaway vServer. if somebody would manage a sandbox escape would have been a really cool but i would definetly not feel comfortable using my own machine after that. yes, everything is nix confed and therefore wipeable+redeployable in theory, that is still not an excuse to potentially give other people knowingly access to my infra.

so: the boring option. tiny attack surface, read-only tools, structural isolation. less capable, but the blast radius of any successful exploit was correspondingly tiny. this felt like the right trade of for a weekend project on personal hardware.

at the event

well, with all of those technicalities one shouldnt forget that my hotline actually needs to get deployed and advertised for users to know it exists at the event.

marketing

now that my hotline was available via 7244 (SAGI on a t9 numpad, meaning Super Artifical General Intelligence of course) my advertisement campaign could begin.

one evening i sat down and designed a one page info flyer of the hotline while sitting in the lounge.

the next issue was that i did not have a printer, but thanks to the friendly people at the poc (phone operation center) who printed my flyer 40 times on the condition that i didnt tell anyone that they had a color printer (oops). after that i found some sticky tape (which was surprisingly harder than expected as everyone was afraid of theirs getting stolen) and i could add my flyers to all the flyer prone walls of the event.

❮

❯

now i was off to waiting for people to call (i had added a tiny vibecoded dashboard to monitor call count, total call time, total agent turn count).

demo

so now finally a demo of me calling in to ask my assistant what to do.

and off i was to the nixos meetup (actually this became one of the highlights of the event for me because i met some really interesting people). without my little assistant i would not have spotted that.

as you can see the latency to first answer is definitely a big issue and especially the first schedule lookup tool call takes really long (~15s). also, of course the agent forgot that it is able to end a phone call itself via tool call.

another note: ironically i still did not get to buying myself a proper dect client, which is why i am calling in via my phone using a sip client app. definitely a thing i need to change, i did not expect to have a dect hotline before having a dect client 😅.

another thing i felt the need of demoing was the queue. basically i just found it funny to hear this song from a low quality dect client speaker (i asked two random people if i could borrow their dect clients for this demo). sorry for the bad audio, the audio file from the microphone seen in the clip got corrupted.

another funny note, i generally tried to pick models that were able to work in multiple languages. this meant that in practice i was able to talk in numerous languages to my assistant. i tried english, german, russian, french and all of them worked. the tts interestingly enough always sounded like the language in which you started the conversation, meaning that if you started in german and then switched to english it sounded like a german person trying really hard to pronounce english words, it was quite funny.

conclusion

the hotline got ~25 calls, ~45 minutes of talk time over the weekend and 8 callers besides me, so not exactly heavy traffic. but it worked: it answered the phone, looked up talks, and pointed me to the nixos meetup, which ended up being one of my highlights of the event. my own hotline giving me a genuinely good recommendation was a nice way for the project to pay for itself.

as expected, the models were the easy part, meaning that once setup they didnt break unexpectedly (although, truth be told, they were a pain to set up). the actual work (and the latency) lived in the harness: turn detection, cancellation, and the path from “caller stops speaking” to first audio. next time that path gets instrumented from day one.

a next iteration might happen 😉 with better latency and hopefully better gpus, which would have made this whole thing a lot more straightforward. and i am still curious about the completely open version where callers can simply instruct the agent to do whatever they want, probably on a rented vm then. a hack the phone box, almost an homage to the old days of phone phreaking.