Brownie — on-device AI on Android

Personal project · Kotlin, Jetpack Compose, LiteRT-LM · source on GitHub

Problem

Every capable AI assistant on a phone is a thin client. The model lives in someone else's datacentre, the conversation travels there and back, and the whole thing stops working on a plane or behind a corporate firewall. Privacy is a policy promise rather than a property of the system.

I wanted to know what was actually left if you refused all of that — not a chatbot demo, but a real agent that calls tools, reads and writes files, remembers across sessions and drives the device, running entirely on the hardware in your pocket.

Constraint

A phone is not a server. There is no meaningful swap, the memory budget is shared with every other running app, and sustained inference runs straight into thermal throttling. The model weights, the tool layer, the conversation history and the UI all have to fit in what a mid-range Android device can spare — and the interface has to stay responsive while the model is working.

The second constraint was self-imposed and harder: no cloud fallback. A hybrid design that quietly calls a remote model when the device struggles would have made every demo smoother and destroyed the reason for building it.

Decisions

• Quantization chosen at runtime, not at build time. The app detects available RAM on first launch and picks a tier: Q4 at roughly 1.5 GB for 6–8 GB devices, Q4 plus a vision projector for 8–12 GB, Q8 with vision above that. Below 6 GB the app says it will not work, rather than failing slowly and blaming the user.
• Five Gradle modules instead of one. :core:inference wraps LiteRT-LM, model download and device detection; :core:agent holds the loop; :core:tools the registry and its twelve implementations; :core:memory persistence, profile and full-text session search; :app the Compose UI. The boundary that earned its keep was inference — it is the piece most certain to be replaced when the next on-device model ships.
• Context compression rather than truncation. When the window fills, the agent summarises and continues. Truncation would have been a day's work; losing the earlier half of a conversation is exactly the failure users cannot diagnose.
• An MCP client. New tools can be added by pointing the app at a server, without shipping an APK.

Result

120 Kotlin files across five modules, 200 commits. Multi-turn tool calling up to ten iterations per message, streaming responses with tool-call chips in the UI, file tools in a sandboxed workspace, contacts, calendar, clipboard, scheduled autonomous tasks, screen reading and device automation through the accessibility APIs, web extraction through a WebView, and speech in both directions.

No cloud, no API key, no data leaving the phone. After the one-time model download, the engine loads in five to fifteen seconds and the app works with the network off.

What I'd do differently

The tool layer outgrew the app. :core:tools ended up larger than the UI module — forty-four files against forty-one — because every new capability meant another implementation compiled into the binary. The MCP client I added in v6 is the right shape, and I would start there: a thin built-in set for the things that genuinely need process access, and everything else behind a tool server that can change without shipping an APK. I reached that boundary late; it should have been the first one.