Himanshu Kumar Singh
All work

Kumbh Mela 2025 · kumbh.gov.in

Kumbh Sahayak

Multilingual assistant for pilgrims, built on hybrid retrieval, tuned prompts and guardrails, running on an 8B open model.

10M+ users · 90%+ accuracy

RAGllama3:8bBGEBM25DSPyTextGradGuardrailsWav2Lip

Overview

Kumbh Sahayak answered pilgrims’ questions about routes, services and rituals across languages during Kumbh Mela 2025. It handled 10k+ concurrent requests at peak. The main constraint was quality on a small open model, so most of the work went into retrieval, prompt optimization and evaluation rather than a bigger LLM.

Architecture

Separate services for the RAG pipeline, guardrails and media generation. Retrieval is hybrid: BM25 for lexical matches and fine-tuned BGE embeddings for semantic ones, followed by a BGE reranker.

  • Hybrid retrieval: BM25 + vector search + BGE reranker
  • Input and output guardrails modelled on AWS Guardrails, plus custom rules
  • Prompt optimization with TextGrad and DSPy against a labelled eval set
  • Fuzzy-match cache (Levenshtein) for repeated questions
  • Similar-chunk merging using Jensen–Shannon divergence
  • Voice cloning (VITS, Coqui-TTS, ElevenLabs) and Wav2Lip lip-sync for official video messages

Query → input guardrails → hybrid retrieval → rerank → LLM → output guardrails → user

What I did

  • Built the question-answering pipeline combining BM25, fuzzy and vector search
  • Fine-tuned BGE embeddings and applied the BGE reranker for ranking quality
  • Optimized prompts with TextGrad and DSPy, measured on a curated evaluation set
  • Designed the safety layer for a public, government-facing deployment
  • Built the speech synthesis and lip-sync pipeline with the Samagra team, including audio clean-up with NVIDIA NeMo

References

  • FlagEmbedding (BGE)

    Embedding and reranker models used for retrieval.

  • TextGrad

    Automatic “differentiation” via text, used for prompt optimization.

  • Wav2Lip

    Lip-sync model from IIIT Hyderabad.

  • VITS

    End-to-end text-to-speech used for voice cloning.