Sunday, July 12, 2026

LawBot Research Paper -- Critique & Analysis -- by Somaiya Vidyavihar University, Mumbai

See All Research on AI Assisted Legal Support Download Research Report

🔍 LawBot (2026) – Critical Review

ML-based chatbot for legal assistance in India • Group 17 • Somaiya Vidyavihar University

👥 Team: Nanda Prem Vasant, Saniya Shah, Ashwin Sinha, Hyder Presswala 📘 Guide: Prof. Chirag Desai Project Report · 2026

1 Architecture

LawBot proposes a pipeline-style architecture with five core modules, designed to process legal queries end‑to‑end:

① Data CollectionScraping + OCR
Indian Kanoon, Supreme Court, Bar Council, user uploads

② Preprocessing & StructuringCleaning + Annotation
Normalise legal text, extract categories, sections, outcomes

③ Model TrainingLegal‑BERT + GPT‑2 + RAG
Fine‑tuned on Indian statutes & case law

④ Query EngineNatural Language → Advice
Retrieval‑Augmented Generation (RAG) with citation‑aware retrieval

⑤ EvaluationAccuracy · F1 · BLEU · ROUGE
Validation on relevance, correctness, and argument quality

The system also includes a citation‑aware component inspired by CiteCaseLAW (2023) and a RAG pipeline that retrieves relevant laws/judgments from a curated Indian legal corpus before generation. A modular design is emphasised for maintainability and future extensibility.

            🔧 Key architectural insight: The combination of Legal‑BERT
            (for retrieval/classification) + GPT‑2 (for generation) + RAG (for grounding)
            is a sound modern pattern for domain‑specific QA, though the paper stops at
            a conceptual blueprint without implementation details.
        

2 Tech‑Stack Currency

The paper references technologies that were state‑of‑the‑art around 2020–2022. Here is a breakdown:

✅ Still relevant

BERT Legal‑BERT – still widely used for legal NLP; fine‑tuning remains effective.
RAG – Retrieval‑Augmented Generation is the dominant paradigm for knowledge‑intensive tasks.
GPT‑2 – while older, it is still a viable lightweight generator; however, GPT‑3.5/4 or Llama‑2/3 would be more performant.
Tesseract / Google Vision API – industry‑standard OCR tools.

⚠️ Outdated / Risky

GPT‑2 (2019) – surpassed by GPT‑3.5, GPT‑4, and open‑source LLaMA‑2/3, Mistral, etc. for fluency and reasoning.
BERT‑base (2018) – while Legal‑BERT is domain‑adapted, more recent models like Legal‑RoBERTa or Case‑Law‑BERT (2023+) offer better performance.
CiteCaseLAW (2023) – the paper uses it as a reference, but the actual model is not integrated; the citation‑aware component is proposed, not implemented.
JSON/XML – standard but not cutting‑edge; modern pipelines often use Parquet or vector DBs (Pinecone, Milvus) for retrieval.

            📉 Verdict: The stack is 2–4 years behind the current frontier.
            For a 2026 project, relying on GPT‑2 and BERT‑base is a significant limitation.
            Modern alternatives like LLaMA‑3, Mistral‑7B,
            or GPT‑4o with fine‑tuning + RAG would yield
            substantially better legal reasoning and fluency.
        

3 Testing & Validation

The paper outlines a validation plan but does not report any actual experimental results from the implemented system. The proposed evaluation metrics are:

Accuracy & F1‑score – for classification tasks (e.g., legal category, violation detection).
BLEU / ROUGE – for measuring the quality of generated legal advice against reference outputs.
Relevance – of the advice provided (84% relevance cited from a 2025 paper, not from LawBot).
Correctness – of law references and citations.

Two test cases are shown (security deposit dispute & unpaid salary) with plausible legal advice, but these are illustrative rather than systematic evaluation results.

            ❌ Critical gap: No quantitative results, no confusion matrix,
            no BLEU/ROUGE scores, no user study, no A/B testing, and no comparison against
            baselines. The paper is a design proposal rather than a
            validated system. The "82.5% accuracy" claim is borrowed from a different
            paper (Garlapati et al., 2025) and not reproduced.
        

4 Datasets

The paper describes intended data sources rather than a curated, released dataset. The planned collection includes:

Indian Kanoon – primary source for judgments and case law.
Supreme Court & High Court portals – official repositories.
Bare Acts & Bar Council portals – statutory texts.
User uploads – PDFs / documents provided by end‑users.
Legal publications – digitised books and journals.

Derived attributes after preprocessing: Legal Category, Summary, Problem Statement, Actionable Steps, and Citations. The dataset is to be stored in JSON/XML format.

            ⚠️ Data concerns:
            No mention of dataset size (number of documents, judgments, tokens).
No licensing / ethics discussion – scraping court portals may have legal
                    restrictions.
No annotation guidelines or inter‑annotator agreement reported.
                
User‑uploaded documents raise privacy & confidentiality risks that are only
                    superficially addressed.

        

5 Plus Points of the Research

            ✔ Social relevance: Addresses a genuine access‑to‑justice gap in India – legal aid
                    is expensive and often inaccessible.
✔ Domain‑specific focus: Tailored to Indian law (Tenancy, Labour, Property) rather
                    than generic legal QA.
✔ Modern architecture: RAG + Legal‑BERT + GPT‑2 is a coherent, production‑ready
                    pattern (even if the chosen models are dated).
✔ Modular design: Clear separation of data collection, preprocessing, training,
                    query engine, and evaluation – good for maintainability.
✔ Citation‑awareness: Attempts to ground advice in actual legal provisions and
                    precedents, which is critical for trustworthiness.
✔ Multimodal input: Supports user‑uploaded documents (PDFs via OCR) – a practical
                    feature for real‑world use.
✔ Non‑functional requirements: Explicitly considers usability, security,
                    portability, and maintainability.

        

6 Gaps & Weaknesses

            ❌ No implementation: The paper is a proposal; there is no working system, no code,
                    no API, no demo.
❌ No empirical evaluation: Zero quantitative results – no accuracy, F1, BLEU, or
                    user satisfaction scores from their own system.
❌ Outdated models: GPT‑2 and BERT‑base are significantly behind current LLMs
                    (GPT‑4, Claude, LLaMA‑3, Mistral).
❌ Dataset undisclosed: No actual dataset is released or described in detail; size,
                    coverage, and quality are unknown.
❌ Legal hallucination risk: No discussion of how to handle incorrect or misleading
                    legal advice – a critical safety issue.
❌ No multilingual support: India has 22 official languages; the system is
                    English‑only, limiting reach.
❌ No real‑time updates: Laws change frequently; no mechanism for keeping the model
                    / corpus current.
❌ Ethics & bias: No analysis of bias in Indian case law or the potential for
                    the system to amplify existing inequalities.
❌ Citation‑awareness is borrowed: The CiteCaseLAW model is referenced but not
                    integrated or adapted.

        

7 Ideas to Learn & Use for "Your Legal"

💡 Actionable takeaways for your project:

RAG is non‑negotiable: Use Retrieval‑Augmented Generation with a vector database (Pinecone, Weaviate, or FAISS) to ground responses in authoritative legal texts.
Choose a modern LLM: Skip GPT‑2; use Llama‑3‑8B, Mistral‑7B, or GPT‑4o (via API) with fine‑tuning on Indian legal data.
Citation‑awareness: Build a retriever that returns relevant sections, acts, and judgments – then force the generator to cite them explicitly.
Modular pipeline: Adopt the same modular structure (collect → preprocess → retrieve → generate → evaluate) – it's a solid blueprint.
User uploads + OCR: Allow users to upload PDFs/ images; use Azure Document Intelligence or Google Document AI for high‑accuracy extraction.
Evaluation first: Build a test suite with gold‑standard Q&A pairs and measure correctness, fluency, and citation accuracy from day one.
Multilingual: Plan for Hindi, Tamil, Telugu, Bengali – use a multilingual embedding model (e.g., LaBSE or mE5) and generate responses in the user's language.
Safety guardrails: Implement a disclaimer, a human‑in‑the‑loop escalation path, and a feedback mechanism to improve over time.
Real‑time updates: Set up a cron job to scrape new judgments from Indian Kanoon / Supreme Court daily and refresh your vector index.

The LawBot paper provides a high‑level roadmap that is conceptually sound. By modernising the model stack, rigorously evaluating, and addressing the gaps, you can build a far more robust and trustworthy system for "Your Legal".

📚 References (from the paper)

Indian Kanoon – primary source for Indian court judgments
Supreme Court of India – official repository
Chalkidis, I. et al., "LawBERT: Pre‑trained Language Model for Legal Text" (2020)
Lewis, P. et al., "Retrieval‑Augmented Generation for Knowledge‑Intensive NLP Tasks" (2020)
CiteCaseLAW – citation‑worthiness detection (2023) – referenced but not integrated
The Indian Contract Act, 1872; Payment of Wages Act, 1936; Transfer of Property Act, 1882
Garlapati, A. et al., "Enhancing Public Access to Legal Knowledge in India" (2025) – cited for baseline accuracy