Open Source Chatbot Resident Services
As a pilot project, IPST developed an open-source chatbot for the City of Bern. Based on RAG technology, it answers questions about legal documents—in compliance with data protection regulations, transparently, and cost-effectively.
Factsheet
- Schools involved Business School
- Institute(s) Institute for Public Sector Transformation
- Strategic thematic field Thematic field "Humane Digital Transformation"
- Duration (planned) 01.09.2024 - 31.01.2025
- Head of project Prof. Dr. Matthias Stürmer
-
Project staff
Luca Sven Rolshoven
Lara Burkhalter - Partner Informatik Stadt Bern
- Keywords Open Source, Chatbot, RAG, Retrieval-Augmented Generation, LLM, Data Protection, Public Sector, Language Model, Haystack, Streamlit, Knowledge Base, City of Bern, Llama, Artificial Intelligence
Situation
The City of Bern needed a solution to efficiently support staff in answering questions about complex legal documents—such as those related to foreign nationals and migration. Existing proprietary AI solutions are limited due to data protection concerns, as sensitive documents cannot be processed on external servers. The Institute for Public Sector Transformation (IPST) at Bern University of Applied Sciences was therefore commissioned to develop a proof of concept (PoC) for a data protection-compliant, transparent, and cost-effective chatbot based on open-source technologies and freely available AI models.
Course of action
The IPST team developed a RAG (Retrieval-Augmented Generation) service that utilizes a knowledge base consisting of 296 documents (3,932 pages, approximately 1.3 million words). Technically, the open-source front-end framework Streamlit and the AI back-end framework Haystack were used. Three language models were used: an embedding model (Snowflake Arctic Embed 2.0), a reranker (BAAI/bge-reranker-v2-m3), and a generative LLM (Llama-3.3-70B-Instruct). Search queries are processed both lexicographically (BM25) and semantically (vector embeddings) and enhanced through contextual retrieval. The chatbot supports follow-up questions and transparently displays sources. The evaluation was conducted qualitatively by a lawyer (Lara Burkhalter) and quantitatively using automated metrics (Mean Average Precision (MAP), Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (nDCG), and Recall for retrieval; Answer Relevancy and Faithfulness for response generation).
Result
In the qualitative evaluation, the chatbot delivers very satisfactory results: responses are largely correct, and the source citations are precise. Particularly noteworthy is its ability to recognize connections between different documents and synthesize them correctly. In the automated evaluation, the selected setup (Top-K Sparse: 20, Top-K Reranker: 7) achieved a recall of 0.73 and an nDCG of 0.46. The LLM used, Llama-3.3-70B-Instruct, achieved an answer relevance of 0.92 and a faithfulness of 0.92. The open-source approach ensures complete data sovereignty, as all data can be processed on the City of Bern’s own servers. The solution is transparent, scalable, and associated with low, predictable operating costs.
Looking ahead
To ensure productive use, the IPST team recommends several further developments: A centralized, categorized list of documents should enable automatic and periodic updates to the knowledge base. In addition, an agent-based workflow is proposed, in which the chatbot can autonomously choose between different tools—such as FAQ queries or restricted web searches. A fundamental prerequisite for reliable operation remains that the database is kept up to date: Outdated documents must be removed regularly, and new ones integrated promptly. Thanks to the modular architecture, new, more powerful AI models can be integrated at any time.
