edge-inferencelisted

# Edge / On-Device Inference ## Why on-device (the decision drivers) - **Latency**: on-device token/inference in ~tens of ms vs 200–500ms cloud round-trip — decisive for voice, AR, control loops. - **Privacy / offline / cost**: data never leaves the device; works without connectivity; no per-call cloud cost. - **Cost of doing it**: tight memory/energy/thermal budgets; smaller models; more engineering. ## The hardware tiers (match model to silicon) | Tier | Silicon | Typical model | |---|---|---| | MCU / TinyML | Cortex-M + tiny NPU (sub-$1 class) | KB-MB models: keyword spotting, anomaly detection (TFLite Micro) | | Mobile / AI-PC | Phone NPU, laptop NPU (tens of TOPS) | Quantized 3–8B LLMs, vision | | Edge box | Jetson Orin/Thor, Coral, Hailo (40+ TOPS) | 7–13B LLMs, multi-camera vision | ## Architectural levers - **Quantization** is the key enabler: FP16 → INT8 → INT4 trades accuracy for memory/throughput/energy. Most edge LLMs run INT4/INT8. Validate accuracy loss. - **Model choice**: small instruct/distilled models (e.g. 3–8B class) and vision models sized to the NPU's memory bandwidth, not just TOPS. - **Runtime**: TFLite/LiteRT, ONNX Runtime, ExecuTorch, llama.cpp/Ollama, vendor SDKs (TensorRT for Jetson). Pick what targets the accelerator. - **Budgets**: state the TOPS, RAM, and energy-per-inference budget up front — they bound everything. Batch=1, KV-cache memory dominates LLM RAM. ## When to recommend edge vs cloud vs hybrid - Hard latency / offline