Building Listen4Me — live captions on-device

Listen4Me was my first iOS app — a small accessibility tool that takes spoken German and prints it onto an iPad or iPhone screen in real time. On-device, no cloud round-trips, German only.

Why this one came first

A relative loses hearing slowly. Restaurants, family dinners, video calls — the same problem each time: she could read lips for a while, then she could not, and the conversation went on without her. Apple's built-in live captions did not exist yet on iOS at the time I started, and the third-party options leaned heavily on cloud transcription. Holding up a phone with someone else's voice running through a server in another country was not the answer.

So the brief was simple. An app that does one thing — listen, transcribe, display the text big — and does it on the device. Something you can hand someone in a noisy room and have it just work.

The first version

v1 of Listen4Me used the built-in SFSpeechRecognizer with on-device recognition flipped on, long-form mode, with a custom audio session that survived headphone toggles and Bluetooth reconnects. The UI was deliberately ugly: a big black screen with white text and a font picker. No tabs. No settings I could avoid.

The harder problem was not transcription. It was state. A hearing-aid user does not want to wonder if the microphone is on. The audio engine has to survive backgrounding for a few seconds and pick up cleanly when the screen lights back up. The text has to keep enough lines on screen to make sense of the last thought without scrolling away too fast. Every one of those tiny decisions was a feedback loop with my relative.

German, on purpose

Listen4Me only does German. The person I built it for speaks German, and the goal was to remove choices, not add them — no language picker, nothing to land on the wrong setting. SFSpeechRecognizer is pinned to German with on-device recognition, and that is the whole language story.

What I would do differently

With today's on-device models, the transcription core would be different. The whisper.cpp work I am doing in Mundwerk for macOS could in principle land on iPhone too — Apple Silicon iPads already have the headroom. A second iteration of Listen4Me would explore that, particularly for the offline case where SFSpeechRecognizer models are not preloaded.

The UI would also get a careful pass. The first version is functional and uncluttered, but it is not beautiful, and accessibility apps deserve the same care as anything else. Typography, contrast, motion reduction — there is room to do better.

Where to read more

The product page is at /apps/listen4me/ and the app sits on the iOS App Store. Listen4Me is a quiet little tool. If you have someone in your life who needs it, that is the kind of audience I had in mind when I built it.