Tsuzi

---

What Tsuzi Can Do Right Now

---

Phase Status

• Phase 1: LangGraph agent foundation
• Phase 2: Long-term memory
• Phase 3: Telegram interface and shared assistant runtime
• Phase 4: Deterministic Windows PC automation
• Phase 5: Screen vision and vision-guided mouse automation

This README documents the project as currently implemented through Phase 5.

---

Experience

You type or send a message
        |
        v
LangGraph agent decides what to do
        |
        +--> calls tools for apps, tasks, mail, reminders, Telegram, PC control
        |
        +--> calls vision tools when UI needs to be located on screen
        |
        v
Tsuzi returns a short assistant response
        |
        +--> optionally speaks it aloud
        +--> logs the full tool trace for debugging

---

Architecture

flowchart TD
    A[Terminal Input / Telegram Message / Telegram Voice] --> B[LangGraph ReAct Agent]
    B --> C[Long-Term Memory]
    B --> D[Wrapped Tools]
    D --> E[Google Calendar / Tasks / Gmail]
    D --> F[Windows Automation]
    D --> G[Telegram Bot]
    D --> H[Screen Vision]
    H --> I[NVIDIA or Ollama Vision Model]
    B --> J[Ollama Text Model]
    B --> K[TTS Output]
    B --> L[Debug Trace Logs]

---

Highlights

Local-first by design

• The main assistant runtime uses Ollama locally.
• Windows automation and screen control happen on-device.
• Screen vision can use local Ollama or NVIDIA API, depending on configuration.
• Voice output is local.

Actually useful beyond chat

• Set alarms and reminders.
• Manage tasks and calendar events.
• Send yourself notifications on Telegram.
• Open apps, switch windows, type text, press shortcuts.
• Ask what is on screen and click UI elements by description.

Debuggable

• Debug mode prints only the important execution trail.
• Every query can be logged with tool calls, results, duration, and final output.

---

Demo Commands

"open chrome and search for LangGraph"
"what windows are open right now?"
"send me a screenshot on Telegram"
"click the search bar and type github"
"read the text on the current screen"
"set an alarm for 7:00 am called workout"
"add a task to finish the README"
"email me the reminder summary"

---

Current Tooling Snapshot

Productivity

• set_alarm
• create_calendar_event
• get_upcoming_events
• add_task
• get_tasks
• complete_task

Communication

• send_email
• read_emails
• search_emails
• Telegram push notifications for reminders and screenshots

PC Automation

• open_app
• close_app
• list_open_windows
• focus_app
• minimize_all
• take_screenshot_tool
• screenshot_to_telegram
• click_at
• type_text_tool
• press_keyboard_key
• hotkey_tool
• control_volume
• control_brightness
• lock_screen_tool
• get_pc_status

Vision Tools

• find_and_click
• find_and_double_click
• find_and_right_click
• find_and_type
• what_is_on_screen
• read_text_on_screen

---

Project Structure

src/
├── main.py                    # terminal + Telegram runtime
├── graph/
│   └── agent.py              # LangGraph agent, prompt flow, debug tracing
├── tools/
│   ├── wrapped_tools.py      # tool registry exposed to the agent
│   ├── google/               # Gmail, Calendar, Tasks integration
│   └── pc_automation/        # Windows automation helpers
├── interfaces/
│   └── telegram_bot.py       # Telegram bot interface
├── services/
│   └── send_reminder.py      # reminder delivery via email + Telegram
├── vision/
│   └── screen_vision.py      # screen understanding and element finding
├── memory/
│   └── long_term_memory.py   # persistent memory storage
└── utils/
    └── config.py             # environment-driven configuration

---

Setup

1. Create the environment

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt

requirements.txt is the source of truth for runtime dependencies. The pyproject.toml file stores project metadata and pytest settings only.

2. Install PyTorch separately

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

3. Configure vision provider

Tsuzi can use NVIDIA for cloud vision and keep Ollama as the local fallback. If VISION_PROVIDER=auto, NVIDIA is used when NVIDIA_API_KEY is set; otherwise the local Ollama vision path is used.

• text model for the assistant runtime: OLLAMA_MODEL
• NVIDIA vision model default: qwen/qwen3.5-122b-a10b
• local Ollama vision fallback default: qwen3.5:4b

Rotate any NVIDIA key that has been pasted into chat or logs before storing it in .env.

4. Configure .env

OLLAMA_MODEL=qwen3.5:4b

VISION_PROVIDER=auto
NVIDIA_API_KEY=your_rotated_nvidia_key
# Leave VISION_MODEL unset to use provider-aware defaults, or override it:
# VISION_MODEL=qwen/qwen3.5-122b-a10b
VISION_TEMPERATURE=0.6
VISION_TOP_P=0.95
VISION_MAX_COMPLETION_TOKENS=16384

# Local Ollama fallback settings
VISION_MODEL_HOST=http://localhost:11434
VISION_NUM_GPU=99

DEBUG_MODE=true

TELEGRAM_BOT_TOKEN=your_bot_token
TELEGRAM_ALLOWED_USER_ID=your_numeric_user_id

GMAIL_ADDRESS=you@gmail.com
GMAIL_APP_PASSWORD=your_app_password
REMINDER_EMAIL=you@gmail.com

5. Run the assistant

python -m src.main

---

Verification

Use the safe smoke gate before opening a PR or starting a new phase:

python -m pytest --collect-only -q --no-test-report
python -m pytest -q -m safe_smoke --no-test-report

The safe smoke suite is deterministic and avoids desktop automation, network, Telegram, Google credentials, Ollama calls, and generated test reports.

Backup artifacts should be compared before deletion. During the readiness audit, flow.ipynb.bak and phase5_plan.md.bak were preserved because they differ from their originals; future *.bak files are ignored.

Manual/local tests are still useful, but some of them may open apps, alter desktop state, use network services, or write ignored files under tests/test_results/. Those tests are marked with side_effect where practical:

python -m pytest
python tests/run_tests.py --quick

For PRs, use the Safe smoke GitHub Actions workflow as the required baseline, then request CodeRabbit/GitHub review for the stabilization PR.

---

Debugging Flow

When DEBUG_MODE=true, Tsuzi prints a concise trace like this:

[debug][terminal_text][a1b2c3d4] Query: open chrome and search for github
[debug][terminal_text][a1b2c3d4] Calling tool: open_app | args: {"app_name":"chrome"}
[debug][terminal_text][a1b2c3d4] Tool result: open_app -> Opened chrome, master.
[debug][terminal_text][a1b2c3d4] Calling tool: find_and_click | args: {"description":"search bar"}
[debug][terminal_text][a1b2c3d4] Final response: Done, master.

Structured query traces are written to:

logs/agent_traces/agent_trace_YYYYMMDD.jsonl

Each record includes:

• query source
• thread id
• tool calls
• tool results
• final response
• duration
• error, if any

---

Vision Notes

Phase 5 is implemented as tool-driven vision, not as a raw multimodal chat UI.

• The text model decides what to do.
• The vision module determines where on screen the target UI element is.
• When NVIDIA vision is active, screenshots are sent to NVIDIA as base64 image inputs. Use VISION_PROVIDER=ollama if screen images must stay local.
• Coordinates are converted back to real screen pixels before mouse automation runs.
• Vision import failures are handled gracefully so the assistant does not crash.

---

Design Choices

Why LangGraph?

It gives you a clean ReAct loop, tool calling, per-thread short-term memory, and a structure that can grow into more autonomous flows without rewriting the assistant from scratch.

Why separate tool wrappers?

The agent sees a compact, well-documented tool interface while the real implementation details stay isolated in service and automation modules.

Why this debugging style?

Raw framework traces are noisy. The current debug path is intentionally narrow: show the query, tool calls, tool outputs, final answer, and keep the rest in JSONL logs for later analysis.

---

Known Scope

• Windows is the primary supported platform right now.
• The assistant is feature-rich but still evolving, so some flows depend on local desktop state and app availability.
• Vision quality depends on the selected NVIDIA/Ollama vision model and the current screen content.

---

Roadmap Context

The broader roadmap continues beyond this point, but the repository currently reflects completed work through Phase 5. The next layers would build on this base rather than replace it.

---

License

MIT