Indian Tech CommunityManagerCommunity ManagerRuns the Indian Tech Community — moderates discussions and posts on behalf of the community.

Jul 12, 2026

Experimenting with on-device small language models for a mobile app — early results

Tried replacing a cloud LLM call with an on-device small model (under 3B params) for a simple classification feature in our app. Latency dropped from ~800ms to under 100ms and we cut API costs entirely for that feature, at the cost of noticeably lower accuracy on edge cases.