Shipping A Model In A Flutter App

Shipping a model in a Flutter app

Flutter is the simplest of the four targets in this suite, for one reason: one package, llamadart, covers both mobile-relevant artifact shapes and every platform Flutter builds for. React Native needs llama.rn for GGUF and react-native-executorch for .pte, two native modules with two build paths. Native Android needs a MediaPipe/LiteRT stack for one shape and a llama.cpp NDK integration for the other. Flutter needs one dependency, one LlamaEngine class, for both.

Check these floors before writing any integration code, as of 2026-07-27: Dart >= 3.10.7, Flutter >= 3.38.0, iOS deployment target 16.4 or newer, macOS deployment target 14.0 or newer (source: llamadart 0.8.17's own installation guide, see the table at the end of this section). llamadart is a young, fast-moving 0.x package, so treat every version number in this skill as an "as of" fact to re-check, not a permanent one. Flutter 3.38.0 and iOS 16.4 are both recent enough to exclude some existing projects; finding that out after wiring up integration code is a wasted afternoon, not a five-minute fix.

Which artifact shape this needs

llamadart 0.8.17 (pub.dev, MIT license, published 2026-07-22) eats both shapes behind the same API. Its own pub.dev description, verbatim: "Dart and Flutter local LLM inference with llama.cpp GGUF and LiteRT-LM across native platforms and web."

| You are holding | llamadart routes it through | What it eats | |---|---|---| | A single .gguf file | llama.cpp | GGUF, self-contained | | A .litertlm file | LiteRT-LM | A LiteRT LM file, produced by litert-torch |

LlamaBackend(), the router passed into LlamaEngine, picks the underlying runtime from the model file's extension: .gguf and unknown extensions go through llama.cpp, .litertlm goes through LiteRT-LM (source: llamadart's own Platform & Backend Matrix doc, "Model format routing"). Both routes share LlamaEngine, ModelSource, ChatSession, and the download and cache APIs; only the load-time parameters that are backend-specific get rejected when used against the wrong format, rather than silently ignored.

If what you are holding is neither of these, a merged Hugging Face checkpoint (config.json + model*.safetensors) or a PEFT adapter directory (adapter_config.json + adapter_model.safetensors), it is not shippable into a Flutter app as is. Run inspecting-a-model-bundle first to confirm which shape you actually have, then convert:

• To GGUF: convert_hf_to_gguf.py then llama-quantize, covered in

references/llamadart-path.md.

• To .litertlm: litert-torch export_hf, also covered there.

A PEFT adapter directory converts to neither format directly. Merge it into its base model first with peft's merge_and_unload(), then export from the merged checkpoint. The GGUF path has a second option: convert the adapter itself with convert_lora_to_gguf.py and load it as a runtime LoRA against a base GGUF, covered in the LoRA section of references/llamadart-path.md.

If it is genuinely unclear which of the two to pick, GGUF is the safer default: it is the higher-coverage artifact across the open-model ecosystem, llamadart documents it as the format with the broader feature surface (embeddings, dynamic LoRA, grammar-constrained decoding, KV-cache state persistence, multimodal projectors), and quantisation tooling for it is mature. Reach for .litertlm when the model is already distributed as a LiteRT-LM bundle, or the target is specifically the Gemma family on Android's GPU or NPU delegate path.

Two other Flutter packages exist and are worth knowing about, not necessarily reaching for. flutter_gemma 1.4.0 (pub.dev, published 2026-07-26) is the MediaPipe/LiteRT lineage alternative: "Run Gemma and other LLMs on-device in Flutter (Android, iOS, Web, Desktop)," eating the same .task / .litertlm shapes covered in shipping-a-model-in-an-android-app. cactus 1.3.0 (pub.dev) is GGUF-based, but its last release was December 2025, seven months stale as of this writing; treat that gap as a caution before depending on it for anything shipping soon, not a recommendation either way.

A word on the web target, since llamadart also reaches it. llamadart compiles into Flutter Web builds (a llama.cpp WebGPU/CPU bridge for GGUF, the @litert-lm/core JavaScript runtime for .litertlm), and that is still Flutter and Dart code, in scope for this skill. A plain JavaScript or React browser app reaching for transformers.js or WebLLM directly is a different stack entirely, and no skill in this suite covers it; that stack is out of scope here and the browser runtimes' own documentation is the place to start.

Install and wiring

# pubspec.yaml
dependencies:
  llamadart: ^0.8.17

For Flutter iOS and macOS builds that should link Apple XCFrameworks through Swift Package Manager, also add the runtime companion packages for whichever backend the app needs:

dependencies:
  llamadart: ^0.8.17
  llamadart_llama_cpp_flutter: ^0.0.11 # GGUF / llama.cpp
  llamadart_litert_lm_flutter: ^0.0.7  # .litertlm / LiteRT-LM on iOS

Then flutter pub get. On the first flutter run or dart run for a native target, llamadart's native-assets hook detects the platform and architecture, resolves matching prebuilt runtime bundles from leehack/llamadart-native and leehack/litert-lm-native, and wires them into the app; no local C++ toolchain setup is needed on the development machine. That first resolution step does reach the network, on the build machine, to fetch the runtime bundle. This is separate from the app's own runtime behaviour: once built, a shipped app loading a model from a local file path does not need network access to generate, which is the property this skill's minimal example is written to demonstrate.

Set the Apple deployment target before running an iOS or macOS build. In Xcode, set IPHONEOS_DEPLOYMENT_TARGET = 16.4 or MACOSX_DEPLOYMENT_TARGET = 14.0 on the relevant Runner configurations; if the iOS project still uses CocoaPods, set the Podfile platform too:

platform :ios, '16.4'

Companion packages, native-assets tag overrides, and per-platform backend trimming (dropping the LiteRT-LM runtime entirely to save package size, for example) are covered in references/llamadart-path.md.

A minimal working example

This adapts llamadart's own quickstart. The package's published quickstart loads from a Hugging Face source (ModelSource.parse('hf://...')) so a first-time reader can copy and paste it without inventing a local path first. That is a fine way to smoke-test the package, but it is a network dependency this skill's own happy path should not carry: a model already trained, exported, and sitting on device should load from that local file directly, with ModelSource.path(...), the local-filesystem constructor confirmed in llamadart's own ModelSource source.

import 'package:llamadart/llamadart.dart';

Future main() async {
  final engine = LlamaEngine(LlamaBackend());

  try {
    await engine.loadModelSource(
      ModelSource.path('/path/to/model.gguf'),
      modelParams: const ModelParams(contextSize: 4096, gpuLayers: 0),
    );

    final messages = [
      LlamaChatMessage.fromText(
        role: LlamaChatRole.user,
        text: 'What is the meaning of life?',
      ),
    ];

    // Streams one chunk per generated delta; each chunk's
    // choices.first.delta.content is the newly generated text fragment,
    // or null on chunks that carry no new text (for example a final
    // finish-reason chunk).
    await for (final chunk in engine.create(
      messages,
      params: const GenerationParams(maxTokens: 256, temp: 0.7),
    )) {
      final text = chunk.choices.first.delta.content;
      if (text != null) {
        // append text to whatever UI state renders the growing response
      }
    }
  } finally {
    await engine.dispose();
  }
}

engine.create(...) is the stateless, chat-template-aware entry point: the model's chat template gets applied, but nothing is remembered between calls, so a follow-up turn means appending both sides of the exchange to messages by hand before calling it again. For a multi-turn chat UI, ChatSession wraps the same engine and keeps history automatically; the trade-off between the two, and ChatSession's own example, are in references/llamadart-path.md. A single, non-streaming full response is the same call awaited to completion rather than iterated chunk by chunk, useful for a backend job that only needs the final text: collect each chunk's choices.first.delta.content into a buffer instead of rendering each piece as it arrives.

gpuLayers: 0 above forces CPU inference, the safest default for a first run on an unknown device. Raise it, or set ModelParams.maxGpuLayers, once GPU acceleration has been confirmed stable for the target hardware; see the platform gotchas below.

The delivery decision

Once the model runs, the next question is how it reaches the device: bundled into the app, or downloaded on first run. Full detail, the app store size ceilings for both platforms, and the general first-run download experience are in references/model-delivery-and-size-budgets.md.

llamadart does not document a package-level bundling cap of its own, the way react-native-executorch's require() route caps at 512 MB. The app store ceilings and install-conversion guidance in the shared reference are what actually bound bundling here, not anything specific to this package. Where llamadart does add something the other three targets in this suite do not have out of the box: loadModelSource(...), backed by DefaultModelDownloadManager and the higher-level ModelDownloadController, is a first-run-download implementation already built into the package, rather than something the app has to assemble from scratch. It reports download progress through the same onProgress callback shown above, resumes a partial download when the server exposes a validator (ETag or Last-Modified) or the caller supplies a SHA-256 checksum, and offers cache policies (preferCached, refresh, cacheOnly, noCache) instead of an app writing that state machine by hand. Of the three things the shared reference says a first-run download needs to get right, real progress, resumability, and something for the user to do while it waits, the first two are largely handled once wired to ModelDownloadController; the third is still a UI decision for the app to make. See references/llamadart-path.md for the download manager's API and its platform-specific default cache directories.

Platform gotchas

• The SDK floors bite before anything else does. Dart >= 3.10.7,

Flutter >= 3.38.0, iOS 16.4+, macOS 14.0+, restated from the top of this skill because it is the single most common way to lose time here: an older project fails to build against llamadart at all, not with a runtime error that points at the cause.

• **The first build reaches the network; the shipped app's inference does

not.** The native-assets hook downloads the platform's runtime bundle from GitHub on a developer's or CI machine the first time a native target builds. That is a build-time dependency on the machine doing the building, separate from what a shipped app needs at runtime to load a local model file and generate. Conflating the two is an easy way to misdescribe this skill's own no-network claim; keep them distinct when explaining it to someone else.

• Runtime LoRA is a GGUF-only feature. setLora(...), removeLora(...),

and adapter stacking are documented as supported on native llama.cpp/GGUF flows. Native LiteRT-LM accepts exactly one default-scale text LoRA adapter at model load, through ModelParams.loras, with no runtime updates, stacking, or custom scaling; LiteRT-LM web does not expose LoRA at all, and WebGPU and LiteRT-LM web both throw an explicit unsupported-operation error for the runtime LoRA API rather than silently no-opping. See references/llamadart-path.md for the full LoRA guide and code.

• **Heavy inference already runs off the UI thread, but streaming

granularity is still a tuning knob.** Native backend operations run in background Dart Isolates by the package's own design, specifically so a generation call does not freeze the Flutter UI. GenerationParams exposes streamBatchTokenThreshold and streamBatchByteThreshold; lower values give more granular, more frequent UI updates per token, higher values reduce isolate message-passing overhead at the cost of chunkier streaming. Default values are reasonable; only change them after measuring a real device.

• Vulkan cooperative-matrix crashes are a driver bug, not a package bug.

Some Vulkan drivers on Windows and Linux advertise cooperative matrix support and then crash inside ggml-vulkan's property queries. The documented workaround is setting GGML_VK_DISABLE_COOPMAT=1 and GGML_VK_DISABLE_COOPMAT2=1 in the process environment before launching; it trades some Vulkan performance for stability and should only be reached for when a Vulkan backend is actually crashing or reporting device loss.

• Package size is configurable, not fixed to "everything included."

Native builds bundle every available runtime family by default, so one build can load both GGUF and .litertlm models. An app that only ever ships one format can trim the other out with llamadart_native_runtimes: [llama_cpp] (or [litert_lm]) under hooks.user_defines.llamadart in pubspec.yaml, per platform if needed. This is a llamadart-specific knob; none of this suite's other three shipping targets expose an equivalent.

• Quantisation floor for GGUF on mobile. Below roughly 3B parameters,

quantising past Q4_K_M is a known quality cliff, not a bug to debug later. Start at Q4_K_M and only go lower after confirming quality still holds.

• Nothing here solves per-device model versioning. If the fine-tune

changes, a bundled asset and a server-hosted download source both need an explicit update path; llamadart's cache metadata tracks whether a given cached file matches its source and checksum, not whether a newer model version exists that the app should switch to.

Full depth on ModelSource variants, the download manager and controller, ChatSession, LoRA, native-assets configuration, and the web build's differences (narrower LiteRT-LM feature surface, crossOriginIsolated and navigator.gpu checks) is in references/llamadart-path.md.

Hand off to

• The artifact shape is not yet confirmed, or it is a merged checkpoint or a

PEFT adapter that needs converting first: inspecting-a-model-bundle

• The model loads but generates badly, repeats, never stops, or the adapter

will not apply: debugging-a-bad-fine-tune

• It is not yet known whether the fine-tune is actually better than the base

model it started from: evaluating-a-tuned-model

• The question is whether running this on device is actually cheaper than an

API at the expected usage level: costing-a-model-vs-an-api

• The target platform is React Native or Expo, native iOS, or native Android

instead of Flutter: shipping-a-model-in-a-react-native-app, shipping-a-model-in-an-ios-app, or shipping-a-model-in-an-android-app

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ErtasAI
• Source: ErtasAI/open-model-skills
• License: Apache-2.0
• Homepage: https://www.ertas.ai

Install and usage instructions live in the source repository linked above.