onnx-genai Knowledge Base
Understand how the runtime is structured, how an inference request moves through it, and which contracts keep execution, memory and plugins correct.
This wiki is written for people first. Each path starts with an explanatory note; links to source and formal documents provide evidence without becoming prerequisite homework.
Choose a learning path
Understand the repository
Start with Repository Map, then follow:
Understand execution providers
Understand memory
Memory Management for Beginners explains allocation, virtual backing, shared mappings, governors, holders and the provider’s stream/context responsibilities from first principles.
MoE Router Skew and Always-On Experts explains how expert-selection skew was measured for a trained-router MoE model, and why it decides whether a residency policy has anything to exploit.
Virtual Memory for KV Cache walks through what the virtual memory manager actually does when a KV cache grows, when a prefix is shared, when one physical handle is mapped at several virtual addresses, and when a release or a teardown synchronization fails.
Understand model prompt formats
Chat Templates explains how a message array becomes the token
sequence a model sees: template structure and conventions, why generation
starts at the position right after the rendered assistant prefix, multimodal
placeholder variants, recipient-scoped channels (to=self / to=user /
to=<tool>), and how tool calls are rendered and fed back.
Chunked Prefill explains why a prompt can be prefilled in pieces without recomputing the KV a chunk attends over, what capping the query width buys, and why the leftover final chunk has to be padded onto a fixed ladder before the kernel cache can stop recompiling the graph on every request.
Change or measure the runtime
- Testing and Verification
- Performance Engineering Playbook
- Chunked Prefill
- Tracing and Profiling
- API Design Principles
Understand contracts and models
Source of truth
Explanations are not specifications
Current code and reproducible measurements take precedence, followed by authoritative documents under
docs/, accepted design decisions, and then these explanatory notes.
The wiki distinguishes shipped behavior from proposed design. If a note and the implementation disagree, treat that as a documentation bug and verify the current source.
Reading in Obsidian
The complete wiki/ directory is an Obsidian-compatible vault. Notes use
stable English paths, YAML properties, wikilinks and callouts while remaining
ordinary Markdown in GitHub and the published site.
See Wiki index and conventions for every note and Using this Wiki for authoring details.