# ExecuTorch on Android: Offline Llama 3.2 Chat and Benchmarks

Author: Meet Prajapati
Published: 2026-11-05
Canonical: https://meetprajapati.com/blogs/executorch-android-offline-llama-chat-benchmarks/

> Build an offline Android chat app with ExecuTorch, Llama 3.2 1B, and Kotlin: Pixel 8a benchmarks, model memory use, and the factual mistakes behind fast replies.

I built a small Android chat app that runs **Llama 3.2 1B Instruct** locally through **ExecuTorch**, with Kotlin and Jetpack Compose for the UI. After copying the model onto the phone, it needs no network connection.

On a Pixel 8a, a short notes summary started generating after about **604 ms** and finished in **3.3 seconds** (warm-run medians). That feels fast. The problem was what it said: one reply changed a deadline, and six attempts at a simple Kotlin expression never produced the right answer.

This post covers the app, how Kotlin reaches the model, the benchmarks, and why a fast reply isn't always a useful one.

## The app

The flow is simple:

1. Copy the compiled model and tokenizer into the app's folder with ADB.
2. Tap **Check again** to load them.
3. Send a message and watch the response stream into the conversation.
4. Ask a follow-up, stop a reply, or clear the chat.

![ExecuTorch AI on a Pixel 8a with Llama 3.2 1B loaded and suggested prompts available](https://meetprajapati.com/blog/executorch-android/model-ready.webp "Figure 1. The model is loaded. The weights live outside the APK, in the app's storage folder.")

Here's an [8.5-second recording of the offline notes-summary flow](https://meetprajapati.com/blog/executorch-android/demo.mp4). It isn't part of the benchmark data: that run shows 657 ms to first output and 60 generated tokens.

The APK has no `INTERNET` permission, and I also ran a reply to completion with Wi-Fi off and Android reporting no active default network. The app doesn't download models. In this demo, copying them over is a developer step.

## Why ExecuTorch and Llama 3.2?

My [llama.cpp summarizer](https://meetprajapati.com/blogs/llama-cpp-android-offline-ai-summarizer-benchmarks/) needed a native build and JNI integration. This time I wanted to try ExecuTorch's packaged Android LLM API: the app calls `LlmModule`, and there's no separate JNI bridge to maintain.

The [official Android LLM guide](https://docs.pytorch.org/executorch/stable/llm/run-on-android.html) is a good place to start, but the API is experimental, so pin a version. This project uses **`org.pytorch:executorch-android:1.5.1`**, and not every snippet you find online will match yours.

I used the community-exported [Llama 3.2 1B Instruct SpinQuant INT4 EO8 model](https://huggingface.co/executorch-community/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8-ET/tree/911221e0f49d3354cbd4697a9c09feee25cadec7). Its `.pte` file is about **1.14 GB**, plus a **2.18 MB** tokenizer. I checked both files against that revision. I didn't export or quantize the model myself.

This is a compiled ExecuTorch program, not a GGUF file. It includes the XNNPACK backend, and the runtime logs show a 2,048-token context and a TikToken tokenizer. Everything here ran on the CPU. I didn't test GPU or NPU inference.

## How a reply reaches Compose

```text
Compose chat screen
  → ChatViewModel
  → ChatRepositoryImpl
  → history pruning + Llama prompt formatting
  → ExecuTorchRunner
  → LlmModule → native runtime
  → callbacks → Kotlin Flow → UI
```

The Gradle dependency is:

```kotlin
implementation("org.pytorch:executorch-android:1.5.1")
```

The rest of the setup is in the [demo project](#user-content-try-the-demo). The runner builds the module from real file paths:

```kotlin
val config = LlmModuleConfig.create()
    .modulePath(modelPaths.modelPtePath)
    .tokenizerPath(modelPaths.tokenizerPath)
    .modelType(LlmModuleConfig.MODEL_TYPE_TEXT)
    .loadMode(LlmModuleConfig.LOAD_MODE_MMAP)
    .temperature(0.7f)
    .build()

val module = LlmModule(config)
module.load()
```

The files live under `getExternalFilesDir(null)`, with internal storage as a fallback. If they're missing, the app shows the ADB commands and a retry button. There's no model importer or downloader.

Generation is a blocking call, so the runner keeps it on `Dispatchers.IO`. Its `LlmCallback` forwards text through `callbackFlow`, and the ViewModel appends each fragment to the current assistant message. The reply streams into the UI while the native call keeps running on its worker thread.

### Format the conversation before generation

The app builds Llama's chat delimiters explicitly. A single-turn prompt looks like this:

```text
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{system prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>

{user message}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

```

For follow-ups, it puts the retained user and assistant messages into the same transcript. The runner resets the native context before replaying that history:

```kotlin
module.resetContext()
module.generate(
    prompt,
    budgetedSeqLen,
    callback,
    /* echo = */ false,
    temperature,
    /* numBos = */ 0,
    /* numEos = */ 0
)
```

This is the overload I used with 1.5.1. Without the reset, the full conversation would be appended to a context that already holds it. The cost is that every follow-up processes the retained history again, because this demo doesn't reuse the previous turn's KV cache.

`budgetedSeqLen` is a budget for the whole sequence, and it doesn't guarantee that a prompt fits. My character-based budgeting still needs work.

## Test setup

| Setting        | Value                                                   |
| -------------- | ------------------------------------------------------- |
| Phone          | Google Pixel 8a                                         |
| SoC / OS       | Tensor G3 / Android 17, API 37                          |
| Build          | Debug APK, min SDK 29, compile/target SDK 35            |
| Runtime        | ExecuTorch Android 1.5.1                                |
| Model          | Llama 3.2 1B Instruct, SpinQuant INT4 EO8               |
| Execution      | CPU; native initialization reports an eight-thread pool |
| Context        | 2,048 tokens, confirmed by runtime metadata             |
| Sampling       | Temperature 0.7, no explicitly fixed seed               |
| Output control | App stops after 512 text callbacks                      |

I ran each of three fixed prompts once right after starting the app, then five more times warm: **18 runs total**. Each request starts with an empty chat, the model stays loaded across warm runs, and I rotated the prompt order between rounds.

The phone was charging with the screen on, and I left at least 45 seconds between runs. Overall thermal status was 0 before each run. I didn't otherwise control background activity, exact temperature, or OS file caching.

## Pixel 8a benchmark results

These are medians of the five warm runs for each prompt:

| Prompt                        | First generated token | Total inference | Decode rate   |
| ----------------------------- | --------------------- | --------------- | ------------- |
| Two-sentence explanation      | 308 ms                | 3,041 ms        | 22.7 tokens/s |
| Three-bullet notes summary    | 604 ms                | 3,279 ms        | 20.1 tokens/s |
| Kotlin expression explanation | 429 ms                | 5,042 ms        | 20.3 tokens/s |

Loading the model took **1,632, 1,428, and 1,538 ms** across the three app restarts. That excludes constructing the module, and restarting the app doesn't flush the OS file cache.

These numbers come from the runtime's own reports, so [its definitions](https://github.com/pytorch/executorch/blob/v1.5.1/extension/llm/runner/stats.h) matter:

* **First-token time** starts when native inference starts, before prompt processing. It doesn't include model loading or the UI work before that call.
* **Total inference includes first-token time.** Don't add the two columns together.
* **Decode rate uses native token counts**, not the UI's callback-based timing.

So these aren't tap-to-result times. ADB typing, Compose rendering, and model loading aren't included.

Response length matters too. The Kotlin runs ranged from **4.2 to 8.3 seconds**, and the longest produced 151 tokens. With temperature 0.7 and no fixed seed, replies vary from run to run, and longer ones take longer even at a similar speed.

Download the [18 runs as CSV](https://meetprajapati.com/blog/executorch-android/benchmark-runs.csv), [exact prompts](https://meetprajapati.com/blog/executorch-android/sample-prompts.txt), [rendered outputs](https://meetprajapati.com/blog/executorch-android/benchmark-outputs.txt), and [setup and metric notes](https://meetprajapati.com/blog/executorch-android/benchmark-notes.txt).

## Fast replies still got facts wrong

### A deadline became a date range

The notes said:

> Ravi will run 12 device tests by November 8.

The first summary replied:

> Ravi will conduct 12 device tests from November 8 to November 15.

November 15 was the planned launch date. The model pulled it into the testing sentence and turned a deadline into a date range.

![The full notes input and three-bullet summary, including the incorrect November 8–15 testing range](https://meetprajapati.com/blog/executorch-android/summary-result.webp "Figure 2. The summary-first run. Its native first-token time was 693 ms; the UI displays 694 ms.")

All six attempts produced three bullets, but five left out that no release had shipped. That's a missing detail rather than a changed fact, so the [manual review](https://meetprajapati.com/blog/executorch-android/quality-review.csv) records it separately.

### A Kotlin explanation skipped the actual filter result

The expression was:

```kotlin
listOf(1, 2, 3).map { it * 2 }.filter { it > 2 }
```

The correct result is `[4, 6]`: mapping produces `[2, 4, 6]`, then filtering removes `2`.

Five of the six replies gave a wrong result. Four returned the unfiltered list, and the first returned `[2, 6, 9]`. The sixth described the steps but never gave a result. One reply even added Python code that would compute `[4, 6]`, with a comment claiming `[2, 4, 6]`.

That's one expression run six times, not a coding benchmark. It still shows that an explanation can sound right while the answer is wrong.

## Multi-turn chat adds work, and can confuse roles

I also ran three separate three-turn conversations. The first turn gave a name, Maya, and an Android project, Cedar. The second asked for both back, and the third added a planned launch date.

In one conversation, the reply said:

> My name is Assistant, and your Android project is called Cedar.

The project name survived, but the model answered about the wrong person. The other two conversations kept the right values, though some replies still said “my name” where they meant “your name.”

Median first-token time rose from **440 ms on turn one** to **573 ms on turn two** and **826 ms on turn three**, as the app replayed more history each time. These turns ran back to back, without the 45-second gaps, so their [nine runs](https://meetprajapati.com/blog/executorch-android/chat-runs.csv) and [outputs](https://meetprajapati.com/blog/executorch-android/chat-outputs.txt) stay separate from the table above.

Before the third conversation, thermal status reached 1, so I paused until it returned to 0. Every recorded turn started at 0. Charging, the screen, and background work could all have contributed, so I can't blame inference alone.

## Memory and long replies need their own checks

Memory mapping the model doesn't make its memory cost go away. Separate `dumpsys meminfo` snapshots showed:

| Stage                                       | App PSS   |
| ------------------------------------------- | --------- |
| App with tokenizer missing                  | 155 MiB   |
| Model loaded                                | 1,294 MiB |
| Largest sampled value during the long reply | 1,382 MiB |

That largest sample is about **1.35 GiB**. It may not be the true peak, and it doesn't tell you whether a low-RAM phone can run the app. Recording and memory polling were on for that run, so its timings stay out of the benchmark table.

A reply can look incomplete for two reasons. The first is a UI problem: a long message runs below the screen, and my auto-scroll stays near its start. Scrolling down by hand reveals the rest of the text and the copy button.

The second is that generation can stop early. In the offline long-reply check, the model promised twelve tips, reached the seventh, and stopped mid-example. It reported 511 generated tokens, which points to the app's 512-callback ceiling, but I didn't log the stop reason, so I can't be certain.

![The end of the long Android-project reply after scrolling, showing an unfinished example and 511 reported output tokens](https://meetprajapati.com/blog/executorch-android/long-reply-bottom.webp "Figure 3. The end is reachable by scrolling, but the generated answer itself is unfinished.")

## Try the demo

The source is on GitHub: [meet30997/executorch-android-offline-chat](https://github.com/meet30997/executorch-android-offline-chat). It includes the app, tests, and Gradle wrapper. The app code matches what I measured, with my local JDK path removed from the build settings. You download the model files separately.

Use **JDK 17** and **Android SDK 35**. Open the project in Android Studio to configure the SDK path, then build and install:

```sh
git clone https://github.com/meet30997/executorch-android-offline-chat.git
cd executorch-android-offline-chat
./gradlew :app:assembleDebug
adb install -r app/build/outputs/apk/debug/app-debug.apk
adb shell am start -n com.example.executorchdemo/.MainActivity
```

Download `Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8.pte` and `tokenizer.model` from the [pinned artifact revision](https://huggingface.co/executorch-community/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8-ET/tree/911221e0f49d3354cbd4697a9c09feee25cadec7), then copy both files:

```sh
adb push Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8.pte /sdcard/Android/data/com.example.executorchdemo/files/
adb push tokenizer.model /sdcard/Android/data/com.example.executorchdemo/files/
```

Tap **Check again** and start with a short English prompt. The [setup notes](https://meetprajapati.com/blog/executorch-android/benchmark-notes.txt) include the tested file hashes. The build packages ARM64 and x86\_64 libraries, but I only tested this model on ARM64, on the Pixel 8a.

## FAQ

### Does this ExecuTorch app work offline?

Yes. Once the model and tokenizer are on the phone, a reply completed with no active default network. The APK also has no `INTERNET` permission.

### Can I load a llama.cpp GGUF model here?

This demo expects the tested ExecuTorch `.pte` program and its matching tokenizer. It doesn't have a GGUF loader.

### How fast is Llama 3.2 1B on the Pixel 8a?

Warm median native first-token times were 308–604 ms across these three short prompts, with total inference medians of 3.0–5.0 seconds. Model loading and UI time are separate.

### Is this faster than the earlier llama.cpp summarizer?

I can't say. The llama.cpp post used a different model, phone, task, and metric definitions, so a fair comparison would need matched conditions.

## Related reading

* [llama.cpp on Android: offline AI summarizer benchmarks](https://meetprajapati.com/blogs/llama-cpp-android-offline-ai-summarizer-benchmarks/)
* [On-device speech-to-text on Android: Parakeet and NeMo benchmarks](https://meetprajapati.com/blogs/on-device-transcription-android-parakeet-nemo-speech/)
* [MediaPipe vs llama.cpp vs ExecuTorch: running AI models on Android](https://meetprajapati.com/blogs/running-on-device-ai-models-android-mediapipe-llamacpp-executorch/)

I'm Meet Prajapati, a Lead Mobile Engineer working on Android and on-device AI. If you're adding offline AI to your app, [get in touch](https://meetprajapati.com/#contact).