How to Set Up AnythingLLM with Ollama

Why AnythingLLM + Ollama Works

AnythingLLM is a full-stack RAG (Retrieval-Augmented Generation) application. It handles document ingestion, chunking, embedding, and retrieval. Ollama is a local LLM runtime: you download a model (Llama2, Mistral, Neural Chat), and it runs inference on your hardware without touching the internet.

Together, they form a complete AI document chat system:

1. You upload documents — PDFs, websites, emails, anything
2. AnythingLLM chunks and embeds — Splits documents into semantic chunks and stores embeddings in Milvus (vector database)
3. You ask a question — Sent to AnythingLLM
4. Semantic search — AnythingLLM finds the most relevant chunks from your documents
5. Local inference — Chunks + question sent to Ollama running on the same server
6. Instant answer — Ollama returns a response, grounded in your data

Nothing leaves your server. No API rate limits. No token counting. No vendor lock-in.

System Requirements

Minimum to run both:

• CPU: 2 cores
• RAM: 4GB (2GB for AnythingLLM + PostgreSQL, 2GB base for Ollama)
• Storage: 20GB for OS and apps, +10GB per LLM model

For better performance with larger models (7B+):

• RAM: 8-16GB
• GPU: Optional but speeds up inference 5-10x (NVIDIA CUDA or AMD ROCm)

Choosing Your LLM Model

Ollama supports hundreds of open-source models. Popular for document chat:

• Llama2 (7B, 13B) — Fast, accurate, widely used
• Mistral (7B) — Lightweight, strong reasoning
• Neural Chat (7B) — Fine-tuned for conversation
• OpenHermes (7B) — Good instruction-following

Download via ollama pull mistral. Takes 5-10 minutes depending on connection.

Multi-Provider Flexibility

Not ready to go fully local? AnythingLLM supports:

• OpenAI (if you have budget)
• Anthropic Claude
• HuggingFace (open-source models)
• Ollama (your choice)

Point to any provider. Your API keys never leave your server. Switch providers without touching your documents.