Jarvis

A personal AI assistant built on a single pydantic-ai agent running a local Ollama model (qwen3:30b). One agent brain, three ways to talk to it — a terminal REPL, a Slack bot, and a fully voice-driven interface — backed by a broad set of tools (calendar, Gmail, tasks, notes, files, the web, your Mac, a browser, Wolfram), human-in-the-loop guardrails on risky actions, and long-term memory.

---

Architecture

Everything is built around one shared Agent defined once in main.py. The three front-ends import that same agent and differ only in how they capture input, render output, and confirm risky actions.

        ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
        │  main.py    │   │ slack_app.py│   │ voice_app.py│   front-ends
        │ (terminal)  │   │  (Slack)    │   │  (voice)    │
        └──────┬──────┘   └──────┬──────┘   └──────┬──────┘
               │                 │                 │
               └────────┬────────┴────────┬────────┘
                        ▼                  ▼
                 ┌──────────────────────────────┐
                 │   agent  (pydantic-ai)        │   one shared brain
                 │   model: ollama qwen3:30b     │
                 │   instructions: prompts.py    │
                 │   + injected date & memories  │
                 └───┬───────────────┬───────────┘
                     │               │
        ┌────────────▼───┐    ┌──────▼─────────────────────────┐
        │  guardrails    │    │  capabilities / tools          │
        │ (gaurdrails.py)│    │                                │
        │ confirm + slow-│    │  web search · web fetch        │
        │ task progress  │    │  MCP servers (below)           │
        └────────────────┘    └──────┬─────────────────────────┘
                                      │
   in-process FastMCP servers ────────┼──────── external MCP servers (subprocess)
   timer · mac · memory · convo       │   gcal · gmail/tasks · files · obsidian
                                      │   slack · time · playwright · wolfram
                                      ▼
            ┌───────────────┐   ┌───────────────┐   ┌───────────────┐
            │  Ollama       │   │  Redis        │   │  SQLite       │
            │  chat + embed │   │  conversation │   │  long-term    │
            │  (host)       │   │  transcript   │   │  facts        │
            └───────────────┘   └───────────────┘   └───────────────┘

Memory (two complementary stores)

• Conversation memory — Redis (convo_store.py): every turn of the live conversation is appended to a Redis list keyed by session. The agent can reach back into it with the convo_recall_conversation tool to answer follow-ups about things said earlier, even after they've scrolled out of the in-process context window.
• Long-term facts — SQLite + embeddings (memory_tools.py): durable facts about you (preferences, names, projects) are stored in SQLite. Each fact is embedded with a local Ollama model (nomic-embed-text); memory_recall ranks stored facts by cosine similarity, so "what units do I like?" finds the fact "I prefer the metric system." The most-recent facts are also injected straight into the system prompt every turn (a "hot" cache), so recent context needs no lookup. The agent saves facts proactively (guided by prompts.py).

Guardrails (human-in-the-loop)

gaurdrails.py installs a tool_execute hook on every tool call:

• Read-only tools (read/list/search/get/…) and an allow-list of benign local tools (LOCAL_SAFE) run without asking.
• Everything else asks for confirmation first; destructive tools (delete/cancel/…) require an explicit "yes". The confirmation is delivered however the active front-end supplies it (terminal prompt, Slack buttons, or voice).
• The same hook emits continuous progress during slow tools when a front-end provides a progress callback — used by voice to say "Working on web search…" and a periodic "Still working…" so it never feels frozen.

Voice pipeline (voice_app.py)

mic ──ffmpeg──▶ WAV ──Groq Whisper (STT)──▶ text ──▶ agent
                                                       │
speakers ◀──afplay──── WAV ◀──Kokoro (TTS)──◀ streamed reply (per sentence)

Confirmations and progress updates are spoken too, so a voice session is fully hands-free.

---

What each file does

File	Purpose
main.py	Builds the one shared agent: picks the Ollama model, assembles all capabilities and MCP servers, registers the instruction injectors (current_date, known_memories), and provides the terminal REPL entry point.
prompts.py	SYSTEM_INSTRUCTIONS — the agent's persona, behavior rules, tool-group overview, and the proactive-memory guidance.
gaurdrails.py	The confirmation/guardrail hook and the Deps dataclass (confirm, progress callbacks). Classifies tools as read-only / destructive / local-safe and wraps slow tools with spoken progress + heartbeat.
memory_tools.py	In-process memory MCP server: remember, recall (semantic), list_memories, forget, backed by SQLite + Ollama embeddings, plus memories_block() (the hot cache injected into context).
convo_store.py	Redis-backed conversation transcript. set_session() / log_turn() helpers used by every front-end, and the convo MCP server exposing recall_conversation.
mac_tools.py	In-process mac MCP server: notifications, volume, clipboard, open app/URL, screenshot — all via macOS built-in CLIs with no shell injection.
timer_tools.py	In-process timer MCP server: countdown timers and clock-time reminders that fire a Mac banner + sound when due.
slack_app.py	Slack front-end (Socket Mode). Routes mentions/DMs to the agent, shows a "thinking" placeholder, and gates risky tools behind Approve / Deny buttons that update in place to show the outcome.
voice_app.py	Voice front-end. Records the mic (ffmpeg), transcribes with Groq Whisper, streams the reply through Kokoro TTS, and handles confirmations + progress by voice.
ollama_patch.py	Small runtime patch so Ollama's OpenAI-compatible endpoint accepts tool-call-only assistant messages (must import before any request).
Dockerfile	Builds the Slack-bot image (Python + Node + uv + Playwright Chromium).
docker-compose.yml	Runs the bot + a Redis service, wired to reach host Ollama.
pyproject.toml / uv.lock	Dependencies. kokoro (voice/TTS, pulls torch) is an opt-in voice extra kept out of the default install and the image.
.env / .env.example	Configuration (tokens, keys, paths). Copy the example to .env.
gcp-oauth.keys.json	Google OAuth client credentials for the Calendar/Gmail/Tasks MCP servers.

Tools available to the agent

Always-on: time, gcal (Google Calendar), google (Gmail + Tasks), timer, mac, memory, convo, plus web search (DuckDuckGo) and web fetch. Loaded on demand via tool search to keep the prompt small: files (filesystem), notes (Obsidian), slack, browser (Playwright), wolfram.

---

Prerequisites

• Python 3.14 and uv
• Ollama running locally with the models:

  ollama pull qwen3:30b          # chat model
  ollama pull nomic-embed-text   # embeddings for semantic memory

• Redis (conversation store): brew install redis && brew services start redis
• Node.js 20+ (npx) — several MCP servers run via npx
• ffmpeg (voice input): brew install ffmpeg
• macOS for the mac/voice features (say/afplay/screencapture are built in). Optional: brew install espeak-ng (Kokoro pronunciation fallback for rare words)
• Accounts/keys as needed: Slack app tokens, Google OAuth client, a free Groq key for voice STT.

Setup

cp .env.example .env          # then fill in tokens/keys/paths
uv sync                       # core install (terminal + Slack)
uv sync --extra voice         # add this if you want the voice interface

# one-time Google Calendar OAuth (opens a browser to authorize)
GOOGLE_OAUTH_CREDENTIALS="$PWD/gcp-oauth.keys.json" \
  npx -y @cocal/google-calendar-mcp auth

Running

Terminal REPL

uv run python main.py
# type messages; "exit" to quit

Slack bot (Socket Mode — no public URL needed)

uv run python slack_app.py
# @mention the bot in a channel, or DM it

Voice (local only — needs mic + speakers)

uv sync --extra voice
uv run python voice_app.py
# press Enter to start talking, Enter again to stop; say "exit" to quit

---

Docker

The container packages the Slack bot (it's headless and long-running; the voice interface needs local audio hardware so it runs on your machine, not in a container). The 30B chat model and embeddings stay in host Ollama — the container reaches them over host.docker.internal.

# with .env filled in and host Ollama + models running:
docker compose up --build

docker compose starts two services:

• redis — the conversation store (data persisted in a named volume)
• jarvis — the bot, built from the Dockerfile, with REDIS_URL, OLLAMA_BASE_URL, and MEMORY_DB_PATH pointed at the compose network / host and a volume so memory.db survives restarts.

Notes:

• Complete the Google OAuth step (above) before containerizing so the tokens exist; ALLOWED_FILE_DIR and OBSIDIAN_VAULT are mounted into the container at the same paths your .env specifies.
• The image deliberately excludes kokoro/torch (voice-only), so it stays lean.

---

Configuration reference

See .env.example for the full list. Key variables:

Variable	Used for
OLLAMA_BASE_URL	Ollama OpenAI-compatible endpoint (chat + embeddings)
EMBED_MODEL	Embedding model for semantic memory (nomic-embed-text)
REDIS_URL	Conversation transcript store
MEMORY_DB_PATH	SQLite long-term memory file (defaults next to the code)
GROQ_API_KEY	Whisper speech-to-text (voice)
STT_MODEL / TTS_VOICE / VOICE_MIC_INDEX	Voice tuning (optional)
SLACK_APP_TOKEN / SLACK_BOT_TOKEN / SLACK_TEAM_ID	Slack bot
GOOGLE_OAUTH_CLIENT_ID / _SECRET / OAUTHLIB_INSECURE_TRANSPORT	Google MCPs
ALLOWED_FILE_DIR / OBSIDIAN_VAULT	Filesystem + notes MCP roots
WOLFRAM_MCP_URL / BROWSERBASE_API_KEY	Wolfram / browser capabilities