If you have ever built a standard, run-of-the-mill vector RAG pipeline only to watch it hallucinate wildly when asked a basic thematic question, welcome to the club. We have jackets, and they are covered in tears and wasted API credits.
Traditional Retrieval-Augmented Generation (RAG) is fantastic at finding a needle in a haystack. If you ask, "What was the exact revenue of our widgets division in Q3?", semantic chunking and vector similarity search will grab the right paragraph and deliver the answer. But if you ask, "What are the main systemic issues across all our quarterly engineering reports?", vanilla RAG falls flat on its face. It is like asking a librarian who only reads index cards to write a thesis on the rise and fall of the Roman Empire—they can find a date, but they cannot synthesise the grand narrative.
Enter GraphRAG by Microsoft (github.com/microsoft/graphrag). This open-source framework completely rethinks how LLMs interact with unstructured data by building a structured, hierarchical knowledge graph before you even run your first query.
Why Vector RAG Fails (And How GraphRAG Rescues It)
Standard RAG treats your documents as an unorganised pile of text snippets. It converts these snippets into vectors, dumps them into a database, and hopes for the best.
GraphRAG, on the other hand, acts like an obsessive detective building a pinboard. It processes your raw data to extract entities (people, places, concepts), identifies the relationships between them, and groups these relationships into hierarchical clusters using community detection algorithms.
| Feature | Naive Vector RAG | Microsoft GraphRAG |
|---|---|---|
| Data Structure | Flat vector embeddings of text chunks | Hierarchical Knowledge Graph (Entities & Relations) |
| Query Strengths | Localised facts, specific keyword/semantic matches | Global syntheses, thematic analysis, multi-hop reasoning |
| Indexing Effort | Low (Fast chunking and embedding) | High (Iterative LLM-based entity extraction) |
| Cost Per Query | Low | Moderate to High (Requires community summary parsing) |
| Hallucination Rate | High on abstract or holistic questions | Low (Grounded in explicit graph relationships) |
The Secret Sauce: Hierarchical Communities
GraphRAG does not just map nodes and edges; it organises them. Utilizing the Leiden community detection algorithm, the framework groups related nodes into "communities" at multiple levels of granularity.
For instance, if your dataset is a collection of news articles, Level 1 communities might represent individual local events. Level 2 might group those events by region, and Level 3 might synthesise them into national geopolitical trends.
When you ask a global query, GraphRAG does not search every raw text chunk. Instead, it queries the pre-generated summaries of these communities, allowing the LLM to synthesise a comprehensive, high-level answer without blowing past your context window limits.
Setting Up GraphRAG Locally
Let’s get our hands dirty. We will set up GraphRAG locally, configure it to run, and index a sample dataset.
Prerequisites
- Python 3.10 to 3.12 installed.
- An API key for OpenAI, Azure OpenAI, or a local alternative like Ollama (though be warned: local models need to be beefy to handle the heavy extraction tasks).
Step 1: Installation
First, create a clean virtual environment and install the package:
python -m venv venv
source venv/bin/activate # On Windows use: venv\Scripts\activate
pip install graphrag
Step 2: Initialize Your Project
Create a directory for your project and run the initialization command. This will generate the necessary configuration files and directory structures.
mkdir -p ./my_graphrag_project/input
cd my_graphrag_project
python -m graphrag.index --init --root .
This generates a settings.yaml file in your root folder. This is where the magic (and configuration) happens.
Step 3: Configure settings.yaml
Open the generated settings.yaml file. You will need to input your API key and define which models you want to use for both generation and embeddings.
encoding_model: cl100k_base
skip_workflows: []
llm:
api_key: ${GRAPHRAG_API_KEY}
type: openai_chat # Or azure_openai_chat
model: gpt-4o
model_supports_json: true
embeddings:
async_mode: threaded
llm:
api_key: ${GRAPHRAG_API_KEY}
type: openai_embedding
model: text-embedding-3-small
Note: Ensure you export your API key in your terminal session before running the indexer:
export GRAPHRAG_API_KEY="your-api-key-here"
Step 4: Add Your Data and Run the Indexer
Drop some unstructured .txt files into the ./input folder. For this test, you could use a public domain novel, a collection of company policy documents, or a batch of transcribed meeting notes.
Now, trigger the pipeline to build your knowledge graph:
python -m graphrag.index --root .
This step will take some time. The pipeline is actively reading your files, prompting the LLM to extract entities and relations, building the graph, clustering it, and writing out summaries. Grab a cuppa.
Querying Your Knowledge Graph
Once the indexing is complete, you can query your data in two distinct ways depending on what you need.
1. Global Search (For high-level, thematic questions)
If you want to know the overarching themes or a summary of the entire dataset:
python -m graphrag.query \
--root . \
--method global \
"What are the primary systemic risks identified in these documents?"
2. Local Search (For specific entity-based questions)
If you want to drill down into a specific person, place, or event and its immediate connections:
python -m graphrag.query \
--root . \
--method local \
"How does Dr. Aris relate to the Project Vanguard initiative?"
Key Takeaways for AI Architects
- Entity-Centric Processing: GraphRAG converts unstructured text into a highly structured web of nodes and edges, preserving context that traditional chunking destroys.
- Hierarchical Summarisation: By clustering nodes and pre-summarising them, the system bypasses the "needle-in-a-haystack" limitation of standard vector databases.
- Cost-Benefit Trade-off: While indexing is computationally expensive and token-heavy up front, it dramatically reduces the cost and improves the quality of complex, high-level queries during runtime.