AI Copilot for Lawyers: A Review of Vector Search, Knowledge Graphs, and a New Approach#

Open‑Source Legal AI Assistants (RAG)#

In recent years, numerous open projects have appeared that showcase copilots for lawyers built on large language models (LLMs) and Retrieval‑Augmented Generation (RAG). As a rule, such systems use vector databases to search legal texts and then generate answers grounded in the retrieved documents. For example, the Legal‑Advisor‑LLM repository describes a chatbot that integrates a vector store for efficient legal information retrieval and combines retrieval with generation to provide contextual responses to legal questions.

Another illustrative example is the Law‑Agent (Adaptive RAG) project. It implements an adaptive RAG algorithm: first, a corpus of legal documents is indexed in a vector store (ChromaDB); then, upon a query, the LLM (GPT‑4) decides whether to look for an answer in the local knowledge base or consult an external source (via web search). If the internal base is sufficient, the model retrieves similar documents by semantic proximity; a specialized module then checks their relevance and quality before passing them as context to the LLM for answer generation.

Beyond English‑language projects, specialized solutions are emerging for other jurisdictions. In China, for example, ChatLaw is being developed for Chinese law. It combines a Mixture‑of‑Experts architecture with knowledge from a legal graph: the developers integrated a graph of relationships among statutes and precedents, as well as manual data moderation, to obtain a high‑quality training dataset. For semantic search, ChatLaw includes a ChatLaw‑Text2Vec module trained on a corpus of Chinese legal texts.

Narrowly focused open‑source tools also deserve mention. The AI Act Law RAG project demonstrates a chatbot for consultations on the European Union's AI Act. The application's backend, using LangChain, creates a vector store for uploaded PDF files of the Act and retrieves relevant fragments, which are then used as context for the LLM when answering the user's questions. As a result, the user receives an answer with direct quotations from the Act, boosting both accuracy and transparency.

As this overview shows, the typical architecture of an open‑source legal copilot looks like this: a corpus of legal texts is pre‑assembled (statutes, codes, court decisions, contracts, etc.), processed, and indexed with a vector database (FAISS, Chroma, Weaviate, and so on). Then, at each query, the system performs semantic search for the closest textual fragments, and finally the language model generates an answer while "reading" those fragments. This approach has already proven its usefulness and is actively developing.

Limitations of Semantic (Vector) Search in Law#

The idea of searching for legal provisions by semantic proximity (vector representations of text) sounds attractive: paired with an embedding model, an LLM can find documents even if the query and the statute are phrased differently. In practice, though, this runs into serious obstacles in law. One reason is the nature of legal language. Legal texts are extremely uniform in style: formal, standardized, and replete with boilerplate. Researchers note that "legal language is often highly repetitive and formulaic," making it difficult for embedding models to distinguish genuinely relevant provisions from merely similar‑sounding text.

A second aspect is the need for contextual (systematic) interpretation of a provision. In legal work it is not enough simply to find an article containing similar words—you must understand how the provision applies within the legal system and to the specific facts. When tackling a matter, a lawyer considers which doctrinal area it falls under, what related provisions must be brought in, and whether there are special conditions or exceptions. This comprehensive analysis is called systematic interpretation, and it requires understanding the provision's place in the legal structure as a whole.

A practical example of the limits of semantic search is given in Microsoft's blog on building a Legal Copilot. In the experiment, the question was: "Water is leaking into my apartment from upstairs. What important case law is there on this problem in Washington State?" It turned out that pure vector search over a database of about 0.5 million cases found only 4 out of 10 truly pertinent decisions (i.e., a recall of around 40%). The remaining six documents surfaced for semantic reasons (they contained words like "water," "apartment," "damage") but were not actually relevant to landlord‑tenant law in Washington State.

Finally, there is a language barrier: many state‑of‑the‑art semantic search models are trained primarily on English texts. For Russian, Kazakh, or, say, French legislation, comparable high‑quality open embedding models may simply not exist. You either need additional training on local legal texts or must use multilingual models with limited accuracy. Thus, trying to locate provisions "head‑on" by semantic similarity alone often fails to deliver acceptable reliability for complex legal tasks.

Knowledge Graphs: An Attempt to Account for Links Among Provisions#

One line of development for legal AI systems is the use of knowledge graphs to represent legal relationships. The idea is that statutes and judicial decisions form a complex network: articles cross‑reference one another; cases establish precedents; concepts are defined in some acts and used in others; and so on. If we explicitly model these relationships as a "node–edge–node" graph (nodes are provisions, cases, concepts; edges are relationships such as "Article A is part of Chapter B" or "Case X cites Precedent Y"), then we can search not just by content similarity but also by structural proximity within the legal system.

Graph technologies are already being applied in legal copilots. Microsoft Research proposed GraphRAG (Graph‑enhanced RAG)—essentially an extension of RAG in which a knowledge graph is automatically extracted from the corpus, and then, at query time, both semantic retrieval and graph querying are used. In the leak example mentioned above, GraphRAG leveraged the case‑citation graph: the corpus encoded which cases are most frequently cited by others on related issues, and this helped identify highly authoritative precedents that pure semantic search had missed.

Another approach to knowledge graphs is the use of ontologies and legal thesauri. For instance, provisions can be linked by legal concepts: specifying that "tenant" and "landlord" are roles within a lease agreement; that an article from the Housing Code belongs to the legal institute of residential lease; and so on. There are academic efforts that extract such entities and relations from statutes and cases and build graph models of the legal domain. In the long run, this could make legal search much more precise and conceptually informed.

In practice, however, knowledge graphs also have limitations. Above all, building a full‑fledged legal graph is hard: it requires expertise and/or advanced NLP. Open projects often simplify the graph to basic links (e.g., citations or cross‑references), which is far from the full spectrum of systematic relationships among provisions. Even in the GraphRAG example, where a precedent network is used, the graph solves only part of the problem—it measures a case's "authority" but does not capture doctrinal nuances or jurisdictional specifics.

This is why graph‑based methods have not become a panacea. They improve retrieval (especially in systems like U.S. precedent law, where citation is highly meaningful), but do not eliminate the need for "understanding." In essence, a knowledge graph is a partial attempt to automate systematic interpretation: to establish a provision's connections to other provisions, to show its place in the system. But a graph cannot effectively and fully "understand" a provision in the sense a competent lawyer does.

A New Approach: A Dynamic Knowledge Graph Built by the LLM Itself#

To conclude, let us outline our original concept, the one we are working on. Having recognized the limitations of both vector search and static graphs, we propose a non‑traditional approach: to build a knowledge graph on the fly, around a specific legal task, using the large language model itself. The idea is that the LLM acts as a kind of "thinking agent" capable of analyzing the query's context and assembling a chain of related legal provisions tailored to that context.

Here is how it works. When a user brings a complex legal question to the copilot, the system does not rush to scour the entire corpus for similar texts. Instead, it engages the LLM in step‑by‑step reasoning: the model forms hypotheses about which legal topics and acts may be relevant, recalls (or clarifies via retrieval) specific provisions to verify, and so on. In effect, the model itself generates a temporary map—a graph of dependencies among provisions, concepts, and precedents—customized to the question at hand.

Crucially, this approach bakes in systematic interpretation of provisions: the LLM "knows" where a provision sits within the legal structure and which related provisions may affect its application. For example, if the question concerns termination of an employee in Kazakhstan, the model will identify the primary act—the Labor Code of Kazakhstan—but will also account for provisions on protections for pregnant employees if pregnancy is mentioned, or provisions of the Civil Code on damages if the question touches on compensation.

How does this differ from a static knowledge graph? In that reasoning by the AI, capable of interpreting context, is in the loop. The graph is not built in advance over the entire dataset, but at query time—taking into account the facts and conditions of the matter. Its nodes are not every "adjacent" article, but only those the model itself has selected as significant after analyzing the problem. The edges in this graph are also determined dynamically: the model might decide that two provisions are connected because they both apply to the current fact pattern, even if they have no obvious formal link.

Our approach resonates with the industry's emerging idea of LLM agents (agentic LLM systems). Experts note that beyond knowledge graphs there is another powerful way to improve copilot accuracy—let the model itself plan and direct retrieval. In such a setup, the LLM can generate clarifying questions, call external tools on its own (a statute database, web search, calculators, etc.), and then integrate the obtained data into the answer. We go a step further: in our system the LLM constructs not just a query plan but an entire conceptual structure—a legal framework—for the problem.

Of course, this approach places high demands on the language model itself—it must have deep domain competence and be able to plan. Yet modern LLMs (especially those trained on legal data or augmented with self‑reflection mechanisms) are increasingly successful at multi‑step tasks. Most importantly, a dynamic, context‑dependent knowledge graph, constructed on the fly, overcomes the constraints inherent to pure vector search and static knowledge bases. The model will not confine itself to articles that "sound similar" but will include all provisions that, given legal logic, should be considered together.

Conclusion#

In sum, modern AI copilots for lawyers are evolving from simple search chatbots into intelligent systems capable of emulating legal reasoning. Open projects have already learned to supply LLMs with up‑to‑date statutes and case law via vector retrieval (RAG). The next step is to teach them to understand the law, not merely to find words. Knowledge graphs and ontologies are attempts to bring the structure of law into algorithms, but without regard to the specific context they are insufficient.

The approach we propose—dynamic construction of a knowledge graph by the LLM itself, targeted at a specific legal problem—opens new possibilities for precision and depth in legal AI. Instead of searching for legal texts by keywords or similarity, such a system would reason about the law, uncovering connections and dependencies that matter for the given case. This could be the key to creating copilots that truly assist lawyers in complex analysis rather than merely automating document search.