Stop shipping RAG chunking regressions.
Compare a pull request's chunk boundaries and retrieval results against a baseline — then fail CI when quality drops.
When a check fails, inspect the exact changed chunks.
✓ 3 strategies benchmarked across 24 test queries.
SplitBench identified 2 semantic break points preserved over fixed-size.
This is a design mockup showing planned functionality, not a working product.
Initial beta focus: Markdown and code documentation in GitHub Actions
Planned compatibility: common vector databases (Pinecone, Weaviate, pgvector, Chroma) and embedding providers (OpenAI, Cohere, local models)
THE PROBLEM
Chunking is the least visible, most breakable part of RAG
Most teams treat document chunking as an afterthought — until chunk regressions and poor top-k recall corrupt retrieval accuracy.
Boilerplate pollution
In some document sets, footer and header text can end up in 40%+ of your chunks, cluttering embeddings so your retriever keeps surfacing junk instead of real content.
Confidently wrong answers
The scariest failures aren't obvious hallucinations — they're answers that look completely correct and only fall apart when you manually check them against the source.
Overlap misconfiguration
Chunk overlap set too low silently kills recall. Teams routinely see major retrieval gains just from tuning overlap from a fixed token count to 20–25% of chunk size — but only after shipping broken retrieval for weeks.
Inconsistent rebuilds
The same document can produce different chunks on every rebuild, and retrieval quality swings with it — with no visibility into what changed.
HOW IT WORKS
From raw docs to a scored comparison in minutes
Diagnose chunk boundaries, optimize overlap ratios, and establish automated pre-production regression testing.
Connect your docs
Upload a sample doc set or connect your existing repository source.
Tune overlap & chunk boundaries
Dynamically compare static fixed-token overlap against 20–25% chunk-size ratios to eliminate dead zones between sentences.
Run every strategy in parallel
Benchmark fixed-size, semantic splitters, and parent-document chunking simultaneously against your eval query set.
Gate your build in CI
Enforce quality baselines. If a rebuild introduces boilerplate bleed or drops recall, SplitBench fails your build automatically.
Overlap Tuning: Fixed Token Overlap vs. Adaptive Ratio Overlap
Static token counts drop cross-chunk semantic continuity, causing vector search to miss multi-sentence answers.
Proportional window preserves complete semantic clauses across chunks, boosting top-k recall on multi-clause queries.
Example result on this test set — actual gains vary by document structure, chunk size, and embedding model.
WHY NOT JUST USE X?
Isn't this just Chonkie / Ragas / ChunkViz?
Short answer: no — but you probably need some of them too. Here's the difference.
| Tool | What it's for | What it doesn't do |
|---|---|---|
Chonkie Chunking Library | Fast, lightweight chunking library with basic visualization | Doesn't diagnose why a specific chunk broke retrieval, doesn't gate in CI on chunk quality regressions |
Ragas / LangSmith / Langfuse Aggregate RAG Evals & Traces | Aggregate RAG evaluation — faithfulness, relevance, recall scores | Not primarily designed to provide versioned, chunk-boundary diffs and merge-blocking checks — focuses on aggregate evaluation and production traces |
ChunkViz Manual Visualizer | Free, manual, one-off visual inspection of chunk boundaries | No regression tracking across rebuilds, no CI integration, no team collaboration |
“Chonkie and Ragas tell you IF your chunks are good in aggregate. SplitBench shows you WHICH specific chunk broke and WHY — and fails your build automatically if it happens again.”
Built for the moment before you deploy
Stop waiting for user complaints or silent hallucinations. Turn chunk quality into a deterministic pass/fail gate in CI.
# Add SplitBench check before vector store indexing
- name: SplitBench Chunk Quality Gate
run: splitbench check --fail-on-regression --baseline=main
env:
SPLITBENCH_API_KEY: ${{ secrets.SPLITBENCH_API_KEY }}
Example output:
ℹ Analyzing 1,420 chunks across 38 markdown files...
✓ Context preservation score: 96.4% (Baseline: 94.0%)
✓ Header/footer noise ratio: 0.8% (Allowed: < 2.0%)
✓ SUCCESS: No chunk regressions detected. Build passing.
Runs as a GitHub Action or CLI step
Designed to embed into your existing workflow in minutes. No major architectural changes required.
Fails the build if chunk quality drops
Automatically compares PR branches against your `main` baseline to catch boilerplate bleed or context fragmentation before merges.
No dashboard-checking required
It's a deterministic pass/fail gate just like your unit tests and linter — actionable in code review comments.
Built by an ML engineer who's shipped local LLM inference pipelines — for anyone building production RAG.
Get early access before public launch
Join early engineers establishing deterministic chunking gates, catching regressions, and preventing silent retrieval failures.
- Help setting up one representative evaluation
- A baseline report on a sample document collection
- Direct feedback with the founder while the workflow is shaped
Everything you need to know before joining
Clear positioning on how SplitBench fits alongside your existing stack, libraries, and eval workflows.