Building an Agentic RAG Pipeline for Legal Question-Answering

Ben Truong

Aug 3, 2025 6 min read Blog

It's hard to keep up with changes in the law, especially if those documents are too long to read.
Why not have an AI read all of them for you - and just prompt control for the other side's (or your own) political bias (or lack thereof!)

Overall Performance & Quality (317 runs)

Latency Waterfall Breakdown (ms)

Total Latency Distribution

Min / Max

22.7s / 135.6s

Median

39.7s

Pipeline Explained

Query Expansion

Expands user's query to consider complementing angles to enrich IR and final answer thoroughness.

Retrieval

BM25 sparse retrieval, then dense retrieval with Qwen3-Embedding-0.6B:F16, and reranking with Qwen3-Reranker-0.6B on RTX 4070 Super.

Synthesis

One-shot agentic answer for top result, then batch calls for reranked docs, followed by a final merge step. Implemented with ChatOllama using gemma3:4b-it-q4_K_M on RTX 3060

Best Answers Showcase

Question: At what age am I protected from age discrimination in employment?

(Score: 5/5 Correct, 5/5 Faithful)

TL;DR: Age discrimination protection under federal law begins at age 40...

High-Consistency Citations

[100.0%] 29 U.S. Code § 623
[100.0%] 29 U.S. Code § 621
[100.0%] 42 U.S. Code § 6103

Recall Success Rate (p=0.9)

BM25 Success Rate: 100.00%
Hybrid Success Rate: 100.00%

Question: Explain what an 'S corporation' is according to the tax code.

(Score: 5/5 Correct, 5/5 Faithful)

TL;DR: An S corporation is a corporation that avoids double taxation...

High-Consistency Citations

No citations met the 90% consistency threshold.

Recall Success Rate (p=0.9)

BM25 Success Rate: 100.00%
Hybrid Success Rate: 100.00%

Question: How does a federal agency formally propose a new regulation?

(Score: 5/5 Correct, 5/5 Faithful)

TL;DR: A federal agency formally proposes a new regulation through a multi-step process...

High-Consistency Citations

[100.0%] 5 U.S. Code § 602

Recall Success Rate (p=0.9)

BM25 Success Rate: 100.00%
Hybrid Success Rate: 100.00%

Question: I am creating a new mobile app. Do I have to make it accessible to people with disabilities?

(Score: 5/5 Correct, 5/5 Faithful)

TL;DR: Yes, you likely have to make your mobile app accessible...

High-Consistency Citations

[100.0%] 47 U.S. Code § 619

Recall Success Rate (p=0.9)

BM25 Success Rate: 100.00%
Hybrid Success Rate: 100.00%

Question: If I donate to a registered charity, what are the rules for getting a tax deduction?

(Score: 3/5 Correct, 5/5 Faithful)

TL;DR: Donating to a registered charity can lead to a tax deduction...

High-Consistency Citations

[100.0%] 119-HR1: One Big Beautiful Bill Act

Recall Success Rate (p=0.9)

BM25 Success Rate: 100.00%
Hybrid Success Rate: 100.00%

Question: If I invent a new type of can opener, how long would a patent protect my invention?

(Score: 5/5 Correct, 5/5 Faithful)

TL;DR: A new type of can opener's patent protection lasts 20 years...

High-Consistency Citations

[95.2%] 35 U.S. Code § 154

Recall Success Rate (p=0.9)

BM25 Success Rate: 100.00%
Hybrid Success Rate: 95.24%

Question: My new employer requires me to fill out an I-9 form. What is the purpose of this form?

(Score: 5/5 Correct, 5/5 Faithful)

TL;DR: The I-9 form is your employer's tool to officially verify that you're authorized to work in the United States.

High-Consistency Citations

[100.0%] 42 U.S. Code § 659
[100.0%] 42 U.S. Code § 653a

Recall Success Rate (p=0.9)

BM25 Success Rate: 100.00%
Hybrid Success Rate: 100.00%

Question: What are my rights to medical leave under the Family and Medical Leave Act (FMLA)?

(Score: 5/5 Correct, 5/5 Faithful)

TL;DR: You are generally eligible for unpaid, job-protected leave under the Family and Medical Leave Act (FMLA) based on your Federal employee status.

High-Consistency Citations

[100.0%] 29 U.S. Code § 2611
[100.0%] 29 U.S. Code § 2613
[100.0%] 29 U.S. Code § 2612

Recall Success Rate (p=0.9)

BM25 Success Rate: 100.00%
Hybrid Success Rate: 100.00%

Question: What are the key provisions of the Secure Act 2.0?

(Score: 5/5 Correct, 5/5 Faithful)

TL;DR: Secure Act 2.0 makes changes to how retirement plans are defined, addresses potential issues with multiple state plan enrollments, and provides funding for implementation.

High-Consistency Citations

[95.5%] 119-HR1: One Big Beautiful Bill Act

Recall Success Rate (p=0.9)

BM25 Success Rate: 100.00%
Hybrid Success Rate: 100.00%

Question: What does the Americans with Disabilities Act (ADA) consider a 'public accommodation'?

(Score: 5/5 Correct, 5/5 Faithful)

TL;DR: A "public accommodation" under the Americans with Disabilities Act (ADA) is essentially any entity...that has complied with its obligations under the law...

High-Consistency Citations

[100.0%] 2 U.S. Code § 1331
[100.0%] 40 U.S. Code § 3314
[100.0%] 29 U.S. Code § 732

Recall Success Rate (p=0.9)

BM25 Success Rate: 100.00%
Hybrid Success Rate: 100.00%

Question: What is the federal minimum wage?

(Score: 5/5 Correct, 5/5 Faithful)

TL;DR: The federal minimum wage has evolved over time...with a current minimum of $7.25 per hour.

High-Consistency Citations

[95.2%] 41 U.S. Code § 6703
[90.5%] 119-HR1: One Big Beautiful Bill Act

Recall Success Rate (p=0.9)

BM25 Success Rate: 95.24%
Hybrid Success Rate: 95.24%

Question: What is the legal definition of a 'security' under federal law?

(Score: 5/5 Correct, 5/5 Faithful)

TL;DR: A “security” under U.S. law is broadly defined as any financial instrument...

High-Consistency Citations

[100.0%] 15 U.S. Code § 77c
[90.5%] 15 U.S. Code § 78c

Recall Success Rate (p=0.9)

BM25 Success Rate: 100.00%
Hybrid Success Rate: 95.24%

Question: What is the main goal of the CHIPS and Science Act?

(Score: 5/5 Correct, 5/5 Faithful)

TL;DR: The CHIPS and Science Act’s core objective is to dramatically boost U.S. semiconductor manufacturing...

High-Consistency Citations

[100.0%] 51 U.S. Code § 70507
[95.5%] 15 U.S. Code § 4603

Recall Success Rate (p=0.9)

BM25 Success Rate: 95.45%
Hybrid Success Rate: 90.91%

Question: What is the process for a work to enter the 'public domain'?

(Score: 3/5 Correct, 5/5 Faithful)

TL;DR: Determining when a work enters the public domain is complex...

High-Consistency Citations

No citations met the 90% consistency threshold.

Recall Success Rate (p=0.9)

BM25 Success Rate: 100.00%
Hybrid Success Rate: 100.00%

Tech DevOps

Building an Agentic RAG Pipeline for Legal Question-Answering

Overall Performance & Quality (317 runs)

Latency Waterfall Breakdown (ms)

Total Latency Distribution

Pipeline Explained

Query Expansion

Retrieval

Synthesis

Best Answers Showcase

High-Consistency Citations

Recall Success Rate (p=0.9)

High-Consistency Citations

Recall Success Rate (p=0.9)

High-Consistency Citations

Recall Success Rate (p=0.9)

High-Consistency Citations

Recall Success Rate (p=0.9)

High-Consistency Citations

Recall Success Rate (p=0.9)

High-Consistency Citations

Recall Success Rate (p=0.9)

High-Consistency Citations

Recall Success Rate (p=0.9)

High-Consistency Citations

Recall Success Rate (p=0.9)

High-Consistency Citations

Recall Success Rate (p=0.9)

High-Consistency Citations

Recall Success Rate (p=0.9)

High-Consistency Citations

Recall Success Rate (p=0.9)

High-Consistency Citations

Recall Success Rate (p=0.9)

High-Consistency Citations

Recall Success Rate (p=0.9)

High-Consistency Citations

Recall Success Rate (p=0.9)

Ben Truong

ML Engineer

Related