Cutting Edge ONNX Embedding Models

A technical analysis based on the ONNX embedding benchmark suite: Github Repo

Abstract

Embedding models map text into dense vectors so that semantic similarity can be measured with cosine similarity or inner product. In retrieval-augmented generation (RAG), those vectors determine which passages reach the language model—and therefore how correct the final answer is.

This article surveys the main types of embedding models represented in this project, explains how they differ in architecture and intended use, and analyzes empirical performance from the checked-in benchmark: nine model families under 1B parameters, exported to ONNX and evaluated at multiple precisions (fp32, fp16, q4). Results show a clear speed–quality frontier: tiny MiniLM models exceed 140 queries per second but score low F1 on the RAG task, while Qwen3-0.6B, ModernBERT-base (q4), and BGE-M3 cluster near ~49–51 F1 at much lower throughput. Quantization is not uniformly harmful; for several families, q4 preserves accuracy and multiplies retrieval speed.

1. Introduction

1.1 Why embedding choice matters

A RAG pipeline typically does four things:

Chunk and embed a document corpus
Index the vectors for nearest-neighbor search
Embed each user query and retrieve top- $(k)$ chunks
Pass retrieved context to a generator LLM

Step 3 is cheap relative to generation, but retrieval quality dominates answer quality. A fast embedder that misses the right passage forces the LLM to invent or fail. A slow, accurate embedder can become a bottleneck for interactive systems or large batch indexing.

This project isolates that tradeoff by fixing the rest of the stack (same documents, same questions, same generator path, same scorer) and varying only the ONNX embedding model and its numeric precision.

1.2 Scope of this repository

Aspect	Detail
Focus	Embedding models under ~1B parameters
Runtime	ONNX Runtime (CPU-oriented deployment path)
Precisions studied	`fp32`, `fp16`, `q4` (additional variants exist on disk)
Evaluation	End-to-end RAG with RAGChecker F1 + retrieval search-time throughput
Corpus	Three mixed-domain documents, 15 evaluation questions

2. What Is an Embedding Model?

2.1 Representation

Given a tokenized sequence $(x = (t_{1}, \dots, t_{n}))$ , an embedding model produces a fixed-length vector $(e (x) \in R^{d})$ . Common pooling strategies include:

CLS token — use the representation at a special classification token
Mean pooling — average last-layer hidden states (optionally mask-aware)
Last-token / EOS pooling — typical for decoder-style embedding models

Similarity is usually $(cos (e (q), e (d)))$ or a scaled inner product after L2 normalization.

2.2 Training objectives (conceptual)

Modern text embedders are rarely plain language-model checkpoints. They are typically trained or fine-tuned with:

Objective family	Idea	Typical products
Contrastive / InfoNCE	Pull query–positive pairs together; push negatives away	MiniLM sentence transformers, many BGE models
Multi-task retrieval	Dense + sparse + multi-vector signals	BGE-M3
Instruction / task prefixes	Condition the vector space on the task (“query: …”, “passage: …”)	EmbeddingGemma, some Qwen3 embedding setups
MLM pretraining then adaptation	Strong bidirectional encoder, then retrieval fine-tune	ModernBERT, classic BERT/BGE

Architecture and training recipe together define the type of embedder more usefully than parameter count alone.

3. A Taxonomy of Embedding Model Types

The models fall into five practical categories. The taxonomy below is intentional for RAG practitioners, not a pure academic genealogy.

Text embedding models (this repo)
├── 1. Compact sentence transformers (MiniLM)
├── 2. Classic English retrieval encoders (BGE-EN)
├── 3. Multilingual multi-function encoders (BGE-M3)
├── 4. Modern long-context bidirectional encoders (ModernBERT)
└── 5. LLM-backbone embedding models
    ├── Decoder-derived (Qwen3-Embedding)
    └── Encoder-adapted generative stacks (EmbeddingGemma)

3.1 Type A — Compact sentence transformers (MiniLM)

Representatives: all-MiniLM-L6-v2, all-MiniLM-L12-v2

Property	L6	L12
Approx. parameters (project reports)	22M	33M
Architecture	BERT-style encoder	BERT-style encoder
Layers	6	12
Hidden size / dim	384	384
Attention heads	12	12
Max position embeddings	512	512
Vocab	30,522 (WordPiece)	30,522

Characteristics

Distilled from larger teacher models; designed for latency and memory.
English-centric sentence similarity heritage (Sentence-Transformers ecosystem).
Short context (512 tokens) is a hard limit for long policy or paper chunks without aggressive chunking.
Small ONNX footprints (tens of MB at q4/int8), easy edge deployment.

When this type wins: high-QPS search, mobile/CPU-only services, prototypes where “good enough” retrieval is acceptable.

3.2 Type B — Classic English retrieval encoders (BGE-EN v1.5)

Representatives: BGE-Base-EN-v1.5, BGE-Large-EN-v1.5

Property	Base	Large
Approx. parameters	143M	~335–572M*
Architecture	BERT	BERT
Layers	12	24
Hidden size	768	1024
Max position embeddings	512	512
Vocab	30,522	30,522

*Project plot metadata lists Large at 572M (consistent with packing/reporting conventions); the underlying Hugging Face BGE-large-en-v1.5 checkpoint is commonly cited near ~335M trainable parameters. Use the project’s relative sizes for comparisons within this benchmark.

Characteristics

Trained specifically for retrieval (query–passage matching), not general NLU.
English-only design; simpler than multilingual multi-function models.
Same 512-token context constraint as MiniLM/BERT.
Base is a middleweight workhorse; Large increases capacity and cost.

When this type wins: English document search with moderate quality needs and BERT-compatible tooling.

3.3 Type C — Multilingual multi-function encoders (BGE-M3)

Representative: BGE-M3 (XLMRobertaModel)

Property	Value
Approx. parameters	541M
Architecture	XLM-RoBERTa encoder
Layers	24
Hidden size	1024
Attention heads	16
Max position embeddings	8194
Vocab	250,002 (SentencePiece, multilingual)

Characteristics

Long context relative to classic BERT (≈8k tokens).
Designed for multi-granularity retrieval (dense embeddings; M3 family also supports sparse and multi-vector modes in the broader BGE-M3 design).
Strong multilingual vocabulary and training—useful beyond English even if this benchmark’s docs are English-heavy.
Large FP32 ONNX size (~2.2 GB with external data) but aggressive q4 shrinks deploy size (~543 MB).

When this type wins: mixed-language corpora, longer chunks, production RAG where quality must stay high under quantization.

3.4 Type D — Modern long-context bidirectional encoders (ModernBERT)

Representatives: ModernBERT-base, ModernBERT-large

Property	Base	Large
Approx. parameters	143M	378M
Architecture	ModernBERT (encoder)	ModernBERT
Layers	22	28
Hidden size	768	1024
Attention heads	12	16
Max position embeddings	8192	8192
Vocab	50,368	50,368
Attention pattern	Hybrid local + global (every $(n)$ layers)	Same family design

Characteristics

A next-generation encoder stack: efficiency-oriented attention (local windows with periodic global layers), RoPE-style position handling, longer native context than BERT-base.
Config shows mean pooling options and masked-LM roots—strong bidirectional understanding before (or alongside) embedding adaptation.
Base punches above its size in this benchmark (especially at q4); Large is slower and, under quantization, more fragile on F1.

When this type wins: English or general long-document retrieval where you want encoder quality without a full 0.5B+ multilingual stack.

3.5 Type E — LLM-backbone embedding models

These models reuse generative language-model backbones, then adapt them for embedding (bidirectional attention flags, pooling, contrastive fine-tuning, task prefixes).

3.5.1 Decoder-derived: Qwen3-Embedding-0.6B

Property	Value
Approx. parameters	572M
Architecture	Qwen3 (transformer LM family)
Layers	28
Hidden size	1024
Attention heads / KV heads	16 / 8 (GQA)
Max position embeddings	32,768
Activation	SiLU
Vocab	~151,669

Characteristics

Longest context window in the suite—important for large sections or lightly chunked docs.
Grouped-query attention (GQA) reduces KV cost relative to full MHA.
Highest fp32 accuracy in the RAG benchmark (50.7 F1).
Slowest class of models at full precision (~6 QPS); q4 roughly doubles throughput with a modest F1 drop.

3.5.2 Generative-stack adapted: EmbeddingGemma-300M

Property	Value
Approx. parameters	294M
Architecture	Gemma3Text with bidirectional attention for embedding
Layers	24
Hidden size	768
Attention heads / KV heads	3 / 1
Max position embeddings	2048
Pattern	Sliding window (512) + periodic full attention
Vocab	262,144

Characteristics

Explicit query/document prefixes in the inference smoke test pattern (task-conditioned embedding).
Hybrid sliding + full attention balances length and cost.
Mid-tier speed (~20 QPS fp, ~36 QPS q4) with mid-low F1 on this particular eval set (~22–24).
Large tokenizer vocab increases embedding table size—relevant for disk and RAM.

When Type E wins: long context (Qwen3), instruction-style retrieval interfaces (Gemma), or alignment with the same model family already used for generation.

4. Precision Types: Another Axis of “Model Type”

Even with one architecture, numeric format changes memory, bandwidth, and latency—and sometimes quality.

Precision	Meaning (practical)	Typical effect in this suite
fp32	Full float32 weights/activations	Baseline quality; largest models; often slower
fp16	Half precision	Smaller weights; throughput often similar to fp32 on CPU (memory-bound vs compute-bound)
q4	4-bit weight quantization (ONNX q4 variants)	Best throughput gains; quality impact model-dependent
Other on disk	`int8`, `uint8`, `bnb4`, `q4f16`, `quantized`	Available for several models; not all appear in the condensed results table

4.1 Observed quantization behavior (high level)

Pattern	Examples
q4 ≈ quality, much faster	BGE-M3 (49.5 → 48.9 F1; ~10 → ~28 QPS)
q4 helps quality or matches best	ModernBERT-base (34.1 fp32 → 50.5 q4); BGE-Base (15.7 → 22.5); EmbedGemma (~22 → 24.4)
q4 hurts quality	Qwen3 (50.7 → 46.8); ModernBERT-large (47.3 → 35.2); MiniLM-L6 (16.7 → 15.6)
fp16 ≈ fp32	BGE-M3, Qwen3, MiniLM-L6 (throughput and F1 nearly tied)

The surprising ModernBERT-base q4 result (F1 higher than fp32 on this small set) should be treated carefully: with only 15 questions, metric variance can be large. Still, it is reproducible in the checked-in numbers and suggests quantization can interact with retrieval ranking in non-monotonic ways—not only “lossy compression.”

5. Benchmark Methodology

Understanding performance requires understanding the metric definitions used here.

5.1 Evaluation pipeline

For each ONNX model variant:

Load tokenizer + ONNX session
Ingest the three source documents into a vector index
For each of 15 questions, search for relevant context
Generate an answer with a fixed OpenAI-compatible LLM
Score with RAGChecker (overall_metrics.f1, plus retriever/generator submetrics in detailed reports)
Record timing fields (ingestion_time_seconds, search_time_seconds, total_eval_time_seconds)

5.2 Metrics reported in `benchmark_results.json`

Metric	Definition in this project
accuracy_f1	RAGChecker overall F1 (%) — answer quality given retrieved context
throughput_qps	$(15/ search_time_seconds)$ — retrieval queries per second

Important caveats:

Throughput measures embedding search / retrieval latency, not full generation E2E latency.
F1 reflects retrieval + generation + scorer interaction; a bad retrieval can still be partially rescued by the LLM, and vice versa.
The eval set is small (n=15) and domain-mixed (policy, governance, systems). Rankings are directional, not MTEB-official leaderboard claims.

5.3 Document mix

The three sources intentionally stress different retrieval styles:

Policy / compliance extraction
Social-science reading comprehension
Technical systems / inference questions

Models that only excel at short English FAQ matching may underperform on longer technical or policy passages—especially if context length is 512 tokens.

All embedding model comparison (F1 v/s throughput)

6. Performance Results

6.1 Full condensed results

Sorted by descending F1, then throughput:

Rank	Model	Precision	Throughput (QPS)	Accuracy F1
1	Qwen3-0.6B	fp32	6.36	50.7
2	ModernBERT-base	q4	13.41	50.5
3	Qwen3-0.6B	fp16	6.17	50.3
4	BGE-M3	fp32	10.02	49.5
5	BGE-M3	fp16	9.73	49.5
6	BGE-M3	q4	27.83	48.9
7	ModernBERT-large	fp32	5.67	47.3
8	Qwen3-0.6B	q4	15.30	46.8
9	ModernBERT-large	q4	8.56	35.2
10	ModernBERT-base	fp32	10.19	34.1
11	EmbedGemma-300M	q4	35.99	24.4
12	BGE-Base-EN-v1.5	q4	68.42	22.5
13	EmbedGemma-300M	fp16	19.87	22.0
14	EmbedGemma-300M	fp32	20.15	21.6
15	all-MiniLM-L12-v2	q4	146.62	20.5
16	all-MiniLM-L12-v2	fp32	92.88	19.3
17	all-MiniLM-L6-v2	fp32	112.57	16.7
18	all-MiniLM-L6-v2	fp16	107.45	16.5
19	BGE-Base-EN-v1.5	fp16	30.77	16.1
20	BGE-Base-EN-v1.5	fp32	31.07	15.7
21	BGE-Large-EN-v1.5	fp32	9.60	15.7
22	all-MiniLM-L6-v2	q4	144.20	15.6
23	BGE-Large-EN-v1.5	q4	20.62	14.5

6.2 Pareto-relevant picks

Goal	Best choice in this suite	Why
Maximum accuracy	Qwen3-0.6B fp32 (50.7 F1)	Top F1; long context
Accuracy with moderate speed	ModernBERT-base q4 (50.5 F1 @ 13.4 QPS)	Near-top F1, ~2× Qwen fp32 throughput
Best quality/speed balance	BGE-M3 q4 (48.9 F1 @ 27.8 QPS)	Near-top quality at ~3× BGE-M3 fp32 speed
Maximum throughput	all-MiniLM-L12-v2 q4 (146.6 QPS)	Fastest; F1 ~20.5 only
Small multilingual-capable deploy	BGE-M3 q4	8k context, strong F1, quantized size

6.3 Performance by model type

Compact MiniLM (Type A)

Dominates the right side of the speed axis (90–147 QPS).
F1 stays in the 15–21 band—lowest quality cluster.
L12 slightly better F1 than L6; L6 slightly faster at fp32 but L12 q4 wins absolute speed.
Interpretation: excellent indexers for high-volume approximate search; weak primary retrievers for this RAG set.

Classic BGE-EN (Type B)

Base q4 is surprisingly competitive on speed (68 QPS) with modest F1 (22.5).
Large is slow and low-F1 here (14.5–15.7)—capacity does not help on this eval; 512-token limit and English-BERT design may not match document length or domain.
Interpretation: English retrieval specialists underperform the modern/multilingual/LLM-embedding cluster on this mixed long-doc task.

BGE-M3 (Type C)

Forms a tight high-quality band (~48.9–49.5 F1).
Quantization is almost free in quality and ~2.8× faster (10 → 28 QPS).
Interpretation: the project’s flagship “balanced production” candidate.

ModernBERT (Type D)

Base q4 is a star outlier (50.5 F1 @ 13.4 QPS).
Base fp32 is much weaker (34.1)—investigate pooling, export, or variance before production trust.
Large fp32 is strong (47.3) but slow; Large q4 degrades sharply (35.2).
Interpretation: architecture is promising; precision selection is critical.

LLM-backbone (Type E)

Qwen3 anchors the accuracy ceiling; use fp16/fp32 for max quality, q4 if you need ~15 QPS.
EmbedGemma sits in a middle band: faster than large encoders, weaker F1 than M3/Qwen/ModernBERT-q4 on this set. Prefix formatting and domain fit may matter more than raw size.

6.4 Parameter count vs quality (project-reported sizes)

Params (M)	Families	F1 band (best precision)
22–33	MiniLM	~16–21
143	ModernBERT-base, BGE-Base	~22–51 (huge spread)
294	EmbedGemma	~22–24
378	ModernBERT-large	~35–47
541–572	BGE-M3, Qwen3, BGE-Large	~15–51

Takeaway: parameter count alone poorly predicts RAG F1. Training objective, context length, and quantization interact more strongly than “bigger is better.” BGE-Large and MiniLM both can land near the bottom of F1 despite very different sizes.

6.5 Context length as a hidden performance factor

Context window	Models
512	MiniLM, BGE-Base/Large EN
2048	EmbeddingGemma
~8192	ModernBERT, BGE-M3
32768	Qwen3-Embedding

On multi-page policy and technical documents, 512-token models rely more on chunking strategy. Long-context models can embed larger coherent units—often improving retrieval of multi-sentence answers. That aligns with the empirical cluster: long-context types dominate the top F1 ranks.

7. Architecture Deep Dive: Why Types Differ

7.1 Bidirectional encoder vs LM backbone

	Classic encoder (BERT / XLM-R / ModernBERT)	LM-backbone embedder (Qwen3 / Gemma)
Attention during embedding	Fully (or hybrid) bidirectional over the sequence	Originally causal; embedding variants enable bidirectional use
Strength	Token-level mutual conditioning; mature pooling recipes	Rich pretraining scale; instruction following; long context
Cost	Scales with layers × seq² (mitigated by local attention)	Often larger vocab and deeper stacks

7.2 Attention efficiency patterns

Full attention every layer: MiniLM, BGE-EN, BGE-M3, Qwen3 — simple, expensive at long $(n)$ .
Hybrid local + global: ModernBERT (global every few layers), EmbeddingGemma (sliding window + periodic full) — better long-sequence scaling.
GQA (grouped query attention): Qwen3 — fewer KV heads reduce memory bandwidth.

7.3 Vocabulary and language coverage

Vocab size	Models	Implication
~30k	MiniLM, BGE-EN	Compact; English WordPiece
~50k	ModernBERT	Modern English-focused tokenizer
~152k	Qwen3	Multilingual generative vocab
~250k	BGE-M3	Strong multilingual coverage
~262k	EmbeddingGemma	Very large embedding matrix

Large vocabs improve tokenization quality for rare words and non-English text but inflate model size (especially embedding tables), which is visible in ONNX disk footprints.

7.4 Task conditioning

EmbeddingGemma’s documented prefix pattern:

query:     "task: search result | query: <text>"
document:  "title: none | text: <text>"

Instruction-style embedders are asymmetric: query and document are not interchangeable. Pipelines must apply the correct prefix on both index and search paths—or quality collapses.

Classic MiniLM/BGE-EN often use simpler symmetric or lightly asymmetric formats. Always match the model card’s recommended template.

8. ONNX Deployment Considerations

8.1 Why ONNX for embeddings

Portable runtime (CPU, some GPU EPs) without full PyTorch
Mature quantization tooling (onnxruntime.quantization, Optimum export)
Predictable ops for production serving

8.2 Approximate on-disk sizes (illustrative)

Model	fp32 order of magnitude	q4 order of magnitude
MiniLM-L6	~87 MB	~52 MB
MiniLM-L12	~127 MB	~59 MB
BGE-Base	~416 MB	~143 MB
ModernBERT-base	~572 MB	~215 MB
EmbedGemma-300M	~1.2 GB	~188 MB
BGE-M3	~2.2 GB	~543 MB
Qwen3-0.6B	~2.3 GB combined	~872 MB
ModernBERT-large	~1.5 GB	~403 MB
BGE-Large	~1.3 GB	~303 MB

Quantization is often the difference between “fits on a laptop edge node” and “needs a dedicated volume.”

9. Practical Selection Guide

9.1 Decision matrix

Scenario	Recommended type	Concrete pick (this benchmark)
Interactive RAG, quality first	LLM-backbone or multi-function	Qwen3 fp32/fp16 or BGE-M3 fp32
Interactive RAG, balanced	Multilingual multi-function @ q4	BGE-M3 q4
Strong quality, modest hardware	Modern encoder @ q4	ModernBERT-base q4 (validate on your data)
High-QPS logging / bulk embed	Compact sentence transformer	MiniLM-L12 q4
Multilingual + long docs	Type C	BGE-M3
Very long documents (>>8k)	Type E decoder	Qwen3-Embedding
English short FAQs only	Type A or B	MiniLM or BGE-Base q4
Strict disk budget	Small + q4	MiniLM q4 or BGE-Base q4

9.2 Anti-patterns suggested by the data

Assuming larger classic BERT is better — BGE-Large underperformed BGE-Base and MiniLM on F1 here.
Deploying only fp32 for “safety” — BGE-M3 loses almost nothing at q4 and nearly triples QPS.
Using MiniLM as sole retriever for hard multi-hop or long-doc RAG — speed is excellent; grounded answer F1 is not.
Ignoring instruction prefixes on Gemma-style models.
Trusting a 15-question ranking as universal truth — always re-eval on your own corpus.

9.3 Suggested production defaults from this suite

Default production: BGE-M3 + q4
Max accuracy offline / low QPS: Qwen3-0.6B + fp16 or fp32
Budget CPU API: ModernBERT-base + q4 after domain validation
Batch preprocessing only: all-MiniLM-L12-v2 + q4

10. Limitations of the Study

Small evaluation set (15 questions) → wide confidence intervals on F1.
Single generator LLM → F1 couples retrieval with that model’s reading behavior.
CPU ONNX path may not mirror GPU TensorRT/CUDA rankings.
Incomplete precision grid — not every model has every precision in the summary JSON.
No pure retrieval metrics in the article summary (e.g., nDCG@k, Recall@k) even though detailed RAGChecker reports may contain retriever-specific fields.
Domain bias toward the three chosen documents.
Parameter counts in plots are report conventions, not always identical to Hugging Face model cards.

11. Related Landscape (Beyond This Repo)

For context, other embedding types practitioners meet outside this project:

Type	Examples	Role
Sparse lexical	BM25, SPLADE	Keyword precision; often hybridized with dense
Multi-vector / late interaction	ColBERT	Higher recall cost; token-level matching
API commercial embedders	OpenAI, Cohere, Voyage	Strong quality; external dependency
Multimodal	CLIP, SigLIP	Image–text spaces (out of scope here)
Code embedders	Voyage-code, StarEncoder	Programming languages

This repository focuses on local, dense, text-only, sub-1B, ONNX models—the set most relevant to self-hosted RAG.

12. Conclusion

Embedding models are not a single product category. In this project they appear as at least five architectural types—compact sentence transformers, classic English retrieval BERTs, multilingual multi-function encoders, modern long-context encoders, and LLM-backbone embedders—further subdivided by precision.

Empirically:

Quality leaders: Qwen3-0.6B (fp32/fp16), ModernBERT-base (q4), BGE-M3 (all precisions).
Speed leaders: all-MiniLM variants (especially L12 q4).
Best overall tradeoff: BGE-M3 q4 (~48.9 F1 at ~28 QPS).
Quantization is a first-class design choice: sometimes nearly free (BGE-M3), sometimes costly (ModernBERT-large, Qwen3), occasionally oddly beneficial on small evals (ModernBERT-base).
Context length and training recipe explain more of the F1 ranking than raw parameter count.

For engineers building RAG systems, the actionable message is simple: type the problem first (language, document length, QPS, disk), then pick an architecture class, then choose the precision that sits on your accuracy–latency Pareto frontier—and re-measure on your documents.

Appendix A — Model Inventory

Local directory	Hugging Face-style identity	Type
`all_MiniLM_L6_v2`	all-MiniLM-L6-v2 ONNX	Compact sentence transformer
`all_MiniLM_L12_v2`	all-MiniLM-L12-v2 ONNX	Compact sentence transformer
`bge_base_en_v1dot5`	bge-base-en-v1.5 ONNX	Classic EN retrieval
`bge_large_en_v1dot5`	bge-large-en-v1.5 ONNX	Classic EN retrieval
`bge_m3`	bge-m3 ONNX	Multilingual multi-function
`modernBERT_base`	ModernBERT-base	Modern encoder
`modernBERT_large`	ModernBERT-large	Modern encoder
`qwen3_embedding_0dot6b`	Qwen3-Embedding-0.6B ONNX	LLM-backbone
`embeddinggemma_300m`	embeddinggemma-300m ONNX	LLM-backbone (Gemma3)

Appendix B — Key Numbers at a Glance

Best F1:          Qwen3-0.6B fp32          50.7
Near-best F1:     ModernBERT-base q4       50.5   @ 13.4 QPS
Balanced:         BGE-M3 q4                48.9   @ 27.8 QPS
Fastest:          all-MiniLM-L12-v2 q4    146.6 QPS  (F1 20.5)
Slowest (fp32):   ModernBERT-large fp32     5.7 QPS  (F1 47.3)

AI Generated Report