Valori Benchmarks
Real workloads. Real containers. Real measurements.
We benchmark Valori using the same engine that powers the product, under explicit CPU and memory limits real measurements from Valori's actual implementation using reproducible workloads, not competitor marketing.
Free-tier reference workload
This is the current validated reference workload for evaluating Valori's Free-tier operating envelope. It is a benchmark configuration, not a production SLA.
The measurements below are a single comparable series 512 MB RAM / 0.5 vCPU throughout from the same benchmark harness (bench_cell.py). Valori's planned Free-tier infrastructure allocation is 1 GB RAM; a separate 1 GB/100K measurement exists and is called out explicitly below rather than merged into this chart, since it was measured under a different container memory limit (see “A note on the 1 GB comparison point”).
- Peak RAM
- ~85.9 MB
- Insert throughput
- ~200 vec/s
- Search p50
- ~116 ms
- Search p95
- ~171 ms
- Restart hash
- PASS
- Peak RAM
- ~349 MB
- Insert throughput
- ~245.8 vec/s
- Search p50
- ~670.7 ms
- Search p95
- ~690.7 ms
- Restart hash
- PASS
- Peak RAM
- ~512 MB (at limit)
- Insert throughput
- ~120 vec/s
- Search p50
- ~1308 ms
- Search p95
- ~1512 ms
- Restart hash
- PASS
Search latency vs. vector count
512 MB RAM / 0.5 vCPU / 384D / BruteForce — the comparable reference series (see Methodology).
Peak memory vs. vector count
512 MB container limit — the 100K bar (amber) reached the container's memory ceiling exactly.
Memory usage becomes the practical constraint as vector count increases. The 100K benchmark remained alive, but its latency degraded substantially and memory reached the 512 MB container limit exactly. 100K is not a recommended Free-tier capacity see “Free-tier capacity” below for the actual engineering recommendation.
Index comparison
Valori currently supports multiple index strategies. We benchmarked them separately to determine which ones provide a meaningful advantage at small managed-instance sizes (1 GB RAM / 0.5 vCPU, 384D, 50K vectors unless noted).
| Index | Search | Recall@10 | Recovery | Current recommendation |
|---|---|---|---|---|
| BruteForce | Predictable | 1.00 | Fast (~4s) | Default |
| IVF | No meaningful latency advantage yet | 1.00 | Higher (17–103s, scales with n_list) | Experimental |
| Binary Quantization | Similar to BruteForce when tuned | 0.99 (tuned) | Fast (~4.2s) | Experimental |
| HNSW | Similar at 10K | Not measured at scale | ~187s (10K) | Not Free default |
- Search
- Predictable
- Recall@10
- 1.00
- Recovery
- Fast (~4s)
- Recommendation
- Default
- Search
- No meaningful latency advantage yet
- Recall@10
- 1.00
- Recovery
- Higher (17–103s)
- Recommendation
- Experimental
- Search
- Similar to BruteForce when tuned
- Recall@10
- 0.99 (tuned)
- Recovery
- Fast (~4.2s)
- Recommendation
- Experimental
- Search
- Similar at 10K
- Recall@10
- Not measured at scale
- Recovery
- ~187s (10K)
- Recommendation
- Not Free default
IVF tuning resultShow detail
We tested n_list and n_probe at 50K vectors / 384D. Search latency stayed around ~660–664 ms across every tested configuration no configuration achieved the 25% p50 improvement we set as the bar for making IVF a default. Changing n_list did materially change recovery time: smaller n_list reduced recovery time substantially.
IVF is implemented and tunable, but current measurements do not show a sufficient search-latency advantage to make it the default managed index.
Binary Quantization tuning resultShow detail
At the tuned configuration: peak memory ≈ 359.6 MB, search p50 ≈ 634 ms, recovery ≈ 4.18s.
BQ can achieve high recall with a larger candidate pool, but the resulting performance advantage over BruteForce is currently too small to justify making it the default. It's useful technology just not yet the default choice for small managed instances.
HNSW measured resultShow detail
In this specific Free-tier reference workload, HNSW did not provide enough search improvement to justify its substantially higher insertion and recovery cost.
Restart integrity
Performance is only useful if the state survives restart correctly. Every benchmark on this page includes a real container restart and a cryptographic state-hash comparison not just a health-check pass.
Earlier in development we found and fixed a case where the state hash could diverge across a restart under specific collection-management operations. It was root-caused, fixed, and is now verified on every benchmark run on this page every result shown here passed this check.
How we benchmark
Environment
Real Docker containers running the actual valori-node release binary, with explicit --memory and --cpus limits (512 MB–1 GB RAM, 0.5 vCPU across the benchmarks on this page).
Dataset
384-dimensional vectors, generated from a fixed random seed (42) for full reproducibility. Every index type in a given comparison is benchmarked against the same generated dataset.
Operations
Insert, search, restart, recovery, and state-hash verification issued over the real HTTP API, not an in-process benchmark harness.
Search
Top-K retrieval (K=10), with p50/p95/p99 latency measured over a sample of real search requests.
Integrity
State hash captured before restart and again after recovery, compared byte-for-byte.
Provenance & reproducibility
The figures on this page are point-in-time results from the open-source harness benchmarks/capacity/scripts/bench_cell.py. They are not a live feed re-run the harness against a valori-node release binary to reproduce or refresh them.
These are point-in-time measurements of the Valori implementation on the configuration described above, not guarantees for every hardware configuration.
Measured vs. validated vs. not yet validated
Directly executed and recorded in a real benchmark run.
Verified repeatedly enough to support an engineering/product recommendation.
Requires more infrastructure or benchmarking before we'll state a number.
| Free-tier reference workload (512MB–1GB / 0.5 vCPU / 384D / BruteForce) | Validated |
| Pro capacity | Not yet validated |
| Enterprise capacity | Not yet validated |
| Storage/disk ceilings | Not yet validated |
| Large-scale concurrency | Not yet validated |
Free-tier capacity
Current testing supports a conservative Free-tier operating envelope of roughly 25K–30K vectors at 384 dimensions using BruteForce on a 1 GB / 0.5 vCPU instance.
This is an engineering recommendation based on measured workloads, not a universal vector-count guarantee. Higher dimensions and different workloads can change the resource profile.
Larger plans
- RAM
- 4 GB
- CPU
- 2 vCPU
- Capacity
- Coming from dedicated validation
- RAM
- 16 GB
- CPU
- 8 vCPU
- Capacity
- Requires dedicated validation
These are planned resource tiers, not validated capacity guarantees. See Pricing for plan details.