On-device generative AI can improve privacy, offline reliability, latency, and inference cost. It also introduces constraints that ordinary network features do not have: compatible hardware, model availability, thermal limits, input limits, and APIs that continue to evolve.
The safest architecture begins with the product capability, not the model name. “Summarize this note privately” is a capability. “Call Gemini Nano” is one implementation.
Choose on-device work deliberately
Local inference is a strong fit when data should remain on the device, the feature must work without a connection, and the task fits the model and token limits of supported devices. Cloud inference remains useful for larger models, wider device reach, and tasks that require more context or stronger reasoning.
Many production apps need a hybrid policy:
- use an on-device API when the capability is available and appropriate;
- fall back to a deterministic local feature when it is not;
- use cloud inference only when the user and product policy allow it.
The user should understand when data leaves the device. Do not silently turn a private local feature into a cloud request.
Put model APIs behind a capability boundary
interface TextAssistant {
suspend fun summarize(
text: String,
options: SummaryOptions,
): SummaryResult
}
sealed interface SummaryResult {
data class Success(val text: String) : SummaryResult
data object Unsupported : SummaryResult
data object InputTooLarge : SummaryResult
data class Failed(val cause: Throwable) : SummaryResult
}This is a small interface with product meaning. It keeps the ViewModel independent from ML Kit, AICore, or a cloud SDK. The implementation can perform capability checks, apply input limits, and translate SDK-specific errors into states the UI can explain.
Make capability a first-class state
Do not assume that every Android device can run the same on-device model. Check support before showing an action as available. Model download or initialization may also take time, so distinguish unsupported, preparing, ready, and temporarily unavailable states.
A lifecycle-aware ViewModel can expose this state through StateFlow. In Compose, collect it with lifecycle awareness and render explicit UI. Avoid launching inference directly from a composable; a recomposition must never start duplicate model work.
Treat inference as cancellable work
Run inference in a structured coroutine owned by the feature’s lifecycle. Cancel obsolete work when the input changes or the user leaves. If the SDK operation cannot be cancelled internally, still prevent stale results from replacing newer state.
fun summarize(text: String) {
summaryJob?.cancel()
summaryJob = viewModelScope.launch {
_uiState.update { it.copy(isLoading = true) }
val result = textAssistant.summarize(
text = text,
options = SummaryOptions.Short,
)
ensureActive()
_uiState.update { current ->
current.withSummaryResult(result)
}
}
}Debounce rapidly changing input where appropriate, limit concurrent requests, and avoid retaining sensitive prompts longer than the feature needs them.
Design for uncertainty
Generated output is not deterministic business data. Validate length and format, give users a way to edit or reject the result, and avoid presenting generated content as verified fact. For high-impact actions, require confirmation and keep deterministic rules outside the model.
Test more than the happy path: unsupported hardware, model unavailable, large input, cancellation, app backgrounding, process recreation, low memory, and malformed output. Record performance and failure categories without logging private prompt content.
A good AI feature still behaves like a good Android feature when the model is unavailable.
