Private inference, RAG, and automation

A private AI stack on a headless RTX 3090 server, reachable only over Tailscale. Local inference, chat UI, retrieval, cross-device memory, and workflow automation, with no cloud LLM involved.

highlights

• Ollama serving 14B to 47B models, tuned to a 32K context with Flash Attention and a q8_0 KV cache (~19 GB of 24 GB VRAM), and load-tested on ~30K-token agent sessions.
• LibreChat in Docker pointed at a custom multi-model Ollama endpoint, with self-hosted SearXNG search, MCP tooling, and MongoDB, Meilisearch, and pgvector backing conversations, search, and RAG.
• A cross-device memory and document service (FastAPI, SQLite FTS5, 768-dim nomic-embed-text embeddings) with reciprocal-rank-fusion hybrid search, an async embedding queue, and an incognito mode. It is exposed to any machine as agent skills.
• An n8n and Postgres email agent that pulls mail from Gmail, classifies it with the LLM, extracts deadlines, and syncs to Google Calendar and Tasks only after approval. A headless Brightspace notifier feeds the same LLM.
• Everything stays on the tailnet. Nothing is exposed to the public internet.

tech