Building On-Device NLP: Privacy, Speed, and the Browser Frontier

When a user speaks to a virtual assistant, the fastest path to a response is one that never leaves the device. On-device NLP eliminates the round trip to a remote server, cutting perceived latency from hundreds of milliseconds to near-zero. More importantly, it means that the words a user types or speaks are processed locally and discarded locally — they are never transmitted, never logged, and never at the mercy of a third-party data retention policy. For applications that handle sensitive domains like healthcare, finance, or personal productivity, this is not a nice-to-have; it is a baseline requirement.

Running inference models in the browser or on a mobile device is genuinely hard. The browser sandbox restricts access to native threads, GPU compute is gated behind WebGL or the still-maturing WebGPU API, and memory budgets are tight. On mobile, you are competing with the operating system, background processes, and thermal throttling. Traditional NLP libraries that were designed for server-side Node.js or Python carry transitive dependencies that balloon bundle sizes past what a mobile WebView can comfortably load. The engineering challenge is not just making a model run — it is making it run fast enough, small enough, and reliably enough that users never notice it is there.

Actor.dev’s NLP engine, @noondra/nlp, was designed from the start with these constraints in mind. It is a pure TypeScript package with a single runtime dependency (budoux, for Japanese word-boundary segmentation). There is no native addon, no WASM blob pulled from a CDN, and no Node.js API surface — it runs identically in V8, JavaScriptCore, and Hermes. The engine handles intent classification, entity extraction, slot filling, and dialogue state management entirely in-process. Model weights are serialized as compact JSON structures that are loaded once and kept in memory for the session, giving deterministic first-response times regardless of network conditions.

The honest trade-off is accuracy versus size. A transformer-based intent classifier will outperform a feature-engineered TF-IDF model on out-of-distribution inputs, but it also costs 50–200 MB of weights that are non-negotiable to download and hold in RAM. For most conversational assistant use cases, the distribution of user inputs is bounded by the domain the assistant covers. A well-trained shallow model with good coverage of that domain beats a large general model that spends its capacity on topics the user will never raise. Actor.dev’s approach leans into this: NLP projects are trained per-agent, the resulting models are small and domain-specific, and accuracy on in-domain inputs is competitive with much larger architectures. When a query genuinely falls outside the model’s confidence threshold, the runtime hands off gracefully to an LLM fallback — keeping the fast path fast while maintaining quality where it matters.