20M Indian Legal Cases Dataset Release

🔑 Enhanced Key Takeaways

• •The dataset addresses a critical data scarcity issue in the Indian legal tech ecosystem, where previously fragmented and non-standardized court data hindered the development of domain-specific Large Language Models (LLMs).
• •The inclusion of a citation graph allows for advanced topological analysis of legal precedents, enabling researchers to map the evolution of Indian jurisprudence and identify 'landmark' cases through network centrality metrics.
• •The project utilizes a hybrid retrieval architecture (Voyage AI + BM25) specifically optimized for the unique linguistic challenges of Indian legal English, which often incorporates archaic terminology and complex procedural syntax.

🛠️ Technical Deep Dive

• •Embedding Model: Voyage AI 'voyage-law-2' (or equivalent domain-specific variant) producing 1024-dimensional dense vectors.
• •Sparse Retrieval: BM25 implementation utilizing custom tokenization rules to handle Indian legal abbreviations and citation formats.
• •Graph Construction: Citation relationships (followed, distinguished, overruled) extracted using a multi-stage pipeline: Regex-based pattern matching for citation strings, followed by LLM-based verification for ambiguous references.
• •Data Pipeline: ETL process handles multi-format source documents (PDF/HTML) from various High Court repositories, normalizing them into Parquet/JSONL formats with standardized schema for judge names, acts, and case outcomes.
• •API Architecture: RESTful interface supporting vector similarity search (k-NN) and metadata filtering.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline