⚡ KARYA Node — Autonomous Local-First Operations Agent

KARYA Node is a Windows-native, local-first autonomous agent for document-heavy government/office workflows. It perceives the screen, drives desktop and browser apps, processes documents, writes code, trains computer-vision models, validates databases, and escalates ambiguous cases to humans — with a full audit trail and a policy guard on every action.

It runs primarily on-device (Gemma via Ollama) and falls back to the Gemini API in the cloud when available.

---

🧭 What's inside

Layer	Tech	Port
Backend API	FastAPI + Uvicorn (async)	8765
Frontend UI	React 19 + Vite	5173
Local LLM	Ollama (Gemma) — optional	11434
CV Training (StratifyLabs)	FastAPI subprocess	8003 (training), 8001 (inference)
DB Validation (ValETL)	FastAPI subprocess	8000

Key modules

• Desktop Agent — screen perception (Gemini Vision) + mouse/keyboard control, runs multi-step tasks autonomously.
• Coding Agent — analyze repos, run terminal commands, tests, and git operations.
• Model Training — natural-language → CV training jobs via StratifyLabs (27 architectures).
• Browser Agent — Playwright-driven web automation.
• Work Queue — durable document-processing pipeline with verification + human escalation.
• Creators Hub / Server Manager / Data Validator — video generation, DevOps, and DB validation bridges.
• Policy Guard + Audit — every action is checked and logged.

---

✅ Prerequisites

• Python 3.12+
• Node.js 20.19+ or 22.12+ (for the frontend)
• A Google API key for Gemini (set in backend/.env)
• (Optional) Ollama with a Gemma model pulled, for fully offline reasoning
• (Optional) Tesseract OCR on PATH for scanned-document extraction

---

🚀 Quick Start

1. Clone & configure

# From the project root
Copy-Item backend\.env.example backend\.env

Edit backend\.env and set your key:

GOOGLE_API_KEY="your-gemini-api-key"
GEMINI_MODEL=gemini-2.0-flash-exp

The .env file is git-ignored. Never commit real keys.

2. Backend

# Create & activate a virtual environment (once)
python -m venv .venv
.\.venv\Scripts\Activate.ps1

# Install dependencies
pip install -r backend\requirements.txt

# (Optional) desktop control + browser automation
pip install pyautogui
python -m playwright install chromium

# Run the API
cd backend
python main.py

The API starts on http://localhost:8765. Check it:

Invoke-RestMethod http://localhost:8765/health

3. Frontend

In a second terminal:

cd frontend
npm install
npm run dev

Open http://localhost:5173. The Vite dev server proxies /api and /ws to the backend automatically, so no extra config is needed.

---

🖥️ Using the Desktop Agent (top feature)

Go to Desktop Agent in the sidebar.

• Run Task — type a plain-language goal (e.g. "Open Notepad and type a leave application"). The agent captures the screen, asks Gemini Vision for the next action, and drives the mouse/keyboard step by step.
• Capture / Analyze Screen — see what the agent sees and get a structured UI analysis.
• Manual Controls — type text, press key combos (ctrl+s, enter, win), and scroll.

⚠️ The agent controls your real mouse & keyboard. Move the mouse to a screen corner to trigger PyAutoGUI's failsafe abort. Requires pip install pyautogui.

---

🧪 Running Tests

cd backend
..\.venv\Scripts\python.exe -m pytest -q

All core + API tests should pass (Ollama/Playwright are optional and degrade gracefully).

---

🎬 Demo Flow

With both servers running, from the Dashboard:

1. Generate Demo Cases — creates synthetic disaster-relief case files.
2. Run Full Demo — enqueues and processes them through the pipeline.

Or via API:

Invoke-RestMethod http://localhost:8765/api/demo/generate-cases -Method Post -ContentType application/json -Body '{"count":5}'
Invoke-RestMethod http://localhost:8765/api/demo/run-full-demo -Method Post

---

🔌 Optional Integrations

These sibling projects launch as subprocesses when you use their features. Each has its own environment:

• StratifyLabs (StratifyLabs/) — CV model training. Started automatically from the Model Training page (training on port 8003, inference on 8001).
• ValETL (valAgent/) — DB validation/migration, used by the Data Validator page (port 8000).

Install their dependencies if you plan to use these features:

# StratifyLabs
cd StratifyLabs; pip install -r requirements.txt
# ValETL (uses uv / pyproject)
cd valAgent; pip install -e .

---

🩺 Troubleshooting

Symptom	Fix
Dashboard shows Offline briefly on load	Normal — it connects within a second via REST polling.
Gemini API: Missing key	Set GOOGLE_API_KEY in backend/.env and restart the backend.
Desktop task does nothing	pip install pyautogui; ensure the target window is visible.
Browser Agent slow on first use	Run python -m playwright install chromium once.
Port already in use	Backend 8765, StratifyLabs 8003/8001, ValETL 8000 — free them or edit the config.
Queue cases fail	Document processing needs Gemini/Ollama reachable; the pipeline still records an auditable failure otherwise.

---

📁 Project Structure

karya-node/
├── backend/            # FastAPI app, agent core, all modules
│   ├── core/           # agent, policy guard, audit, config, verifier
│   ├── modules/        # desktop, coding, browser, model_training, valetl, ...
│   ├── data/           # synthetic cases, queue, outputs
│   └── main.py         # API entry point
├── frontend/           # React + Vite UI
├── StratifyLabs/       # CV training platform (optional subprocess)
└── valAgent/           # ETL/DB validation (optional subprocess)

---

🔒 Security Notes

• Every agent action passes through the Policy Guard (destructive-command blocklist, path/domain allowlists, approval gates for deletes & external submits).
• All actions are recorded in an audit log viewable on the Audit Timeline.
• Keep backend/.env out of version control (already git-ignored).