File Manager

A modern web-based file management application built with Flask. This application provides a user-friendly interface for browsing, searching, and managing files in a specified directory, with support for metadata tracking, filtering, and sorting.

Features

• File Management

  • Scan and index files from a specified directory
  • Track comprehensive file metadata (creation, modification, and addition dates)
  • Support for images, videos, and other file types
  • Secure file serving with proper MIME types
  • Video streaming with range request support

• AI Image Processing (OCR + Description)

  • Per-image OCR text extraction and 2-4 sentence description via a local vision-language model (VLM) running on Ollama on your LAN.
  • Job queue persisted in SQLite, managed entirely from the web app.
  • Each run records the model, host, prompt, output, timing, and attempt count so you can compare runs across model upgrades.
  • /processing dashboard: queue counts by task/status, paginated and sortable jobs table with per-job filters (task, status, model, filename), host/model breakdown, and bulk actions.
  • Worker controls: Pause/Resume the worker (state persists across restarts), Cancel pending, Retry failed, Enqueue-all, and a Clear dropdown to delete runs by status (or all non-running).
  • Per-job actions: cancel individual pending runs, retry individual failed/cancelled runs.
  • File modal shows the latest OCR text and description with a per-file Reprocess button.
  • Background worker reaches out to Ollama over the LAN; configurable model and timeouts via environment variables.

• Advanced Filtering & Sorting

  • Filter by file type (images, videos, other)
  • Filter by AI processing state ("Has OCR" / "Has Description", tri-state)
  • Sort by OCR Updated / Description Updated (the finished_at of the file's current OCR or VLM run)
  • Date range filters for creation, modification, and addition dates
  • Time period quick filters (today, week, month)
  • File size range filter with interactive slider
  • Tag-based filtering
  • SQLite FTS5 full-text search across filenames, notes, OCR text, and AI descriptions
  • Multi-column sorting with ascending/descending options

• User Interface

  • Responsive grid layout
  • File cards with previews and metadata
  • Modal viewer for images, videos, and PDFs
  • Collapsible filter sections
  • Smart pagination
  • Real-time statistics

• File Operations

  • View files in browser
  • Download files
  • Add/edit notes
  • Manage tags
  • Update file metadata

Architecture

Backend

• Flask: Web framework
• SQLAlchemy: ORM for database operations
• SQLite: Database (bind-mounted data/ directory; WAL mode, FTS5)
• Pillow / FFmpeg: Image and video thumbnails
• gunicorn: WSGI server (single worker process; the background worker's run-claim is not multi-process safe)

Frontend

• Bootstrap 4: UI framework
• jQuery: DOM manipulation and AJAX
• noUiSlider: Range slider for file size filtering
• Font Awesome: Icons

Project Structure

/
├── app/
│   ├── __init__.py      # App initialization, schema migration, worker startup
│   ├── models.py        # Database models (File, Tag, ProcessingRun, Setting)
│   ├── routes.py        # Route handlers (incl. /processing endpoints)
│   ├── settings.py      # Configuration (incl. Ollama / worker settings)
│   ├── worker.py        # Background OCR/VLM worker (calls Ollama over LAN)
│   ├── fts.py           # SQLite FTS5 full-text search index helpers
│   ├── static/js/       # Frontend JS
│   └── templates/
│       ├── base.html       # Layout + nav
│       ├── index.html      # File browser
│       ├── processing.html # Processing dashboard
│       └── api.html        # API documentation
├── storage/            # File storage directory (thumbnails/ lives inside it)
├── data/               # SQLite database (bind-mounted)
├── requirements.txt    # Python dependencies
├── test_smoke.py       # End-to-end smoke test (run with: python test_smoke.py)
├── .env.example        # Sample environment configuration
├── docker-compose.mbp.yaml        # Local (Mac) Docker Compose
├── docker-compose.synology.yaml   # Synology NAS Docker Compose
└── README.md           # This file

Database Schema

File Model

• id: Integer (Primary Key)
• filename: String (Secure filename)
• original_filename: String
• file_type: String (image, video, other)
• mime_type: String
• size: Integer (bytes)
• created_at: DateTime (File creation time)
• last_modified_at: DateTime (File modification time)
• added_at: DateTime (When added to DB)
• updated_at: DateTime (DB record update time)
• notes: Text
• current_ocr_run_id: FK to most recent successful OCR run (nullable)
• current_vlm_run_id: FK to most recent successful VLM run (nullable)
• tags: Many-to-many relationship with Tag

Tag Model

• id: Integer (Primary Key)
• name: String (Unique)
• created_at: DateTime

Setting Model

Tiny key/value store for runtime app settings (e.g. the worker pause flag).

• key: String (Primary Key) — e.g. worker_paused
• value: Text
• updated_at: DateTime

ProcessingRun Model

Tracks each OCR or VLM processing attempt for a file. History is preserved across re-runs so you can compare outputs from different models.

• id: Integer (Primary Key)
• file_id: FK to File
• task: String (ocr | vlm)
• status: String (pending | running | done | failed | cancelled)
• model: String (e.g. qwen3.6:35b)
• host: String (which machine ran the job)
• prompt: Text (prompt used for this run, captured for auditability)
• output: Text (model output)
• error: Text (error message if failed)
• attempt: Integer
• started_at, finished_at, duration_ms, created_at

The (status, task) and (file_id, task) indexes keep the worker poll and per-file history lookups fast.

Deployment

Prerequisites

• Docker and Docker Compose
• Git

Quick Start

1. Clone the repository:

   git clone https://github.com/mcotton/listing.git
   cd listing
   

2. Create the database directory (file storage is bind-mounted from your existing screenshots/recordings directory — see the compose file):

   mkdir -p data
   

3. Set up environment variables:

   cp .env.example .env
   # Edit .env with your configuration
   

4. Start the application:

   docker compose -f docker-compose.mbp.yaml up -d
   

5. Access the application at http://localhost:5005 (Synology: port 9008)

Docker Compose Configuration

Two compose files are provided:

• docker-compose.mbp.yaml — local development on a Mac. Uses host.docker.internal for Ollama by default.
• docker-compose.synology.yaml — Synology NAS deployment. Set OLLAMA_URL to the LAN address of your Mac/CUDA workstation running Ollama.

(An untracked docker-compose.yaml can be used as a local override.)

The container runs gunicorn with a single worker process — the background worker's run-claim is not multi-process safe. Threads handle request concurrency.

See the file contents for the full set of environment variables passed to the container.

Environment Variables

Copy .env.example to .env and edit as needed.

Core:

• FLASK_APP: Application entry point (default: app)
• FLASK_ENV: Environment (development/production)
• STORAGE_PATH: Path to file storage directory
• THUMBNAILS_PATH: Path to thumbnail cache
• DATABASE_URL: SQLAlchemy DB URL (code default sqlite:///files.db; the compose files set sqlite:////app/data/files.db — note the four slashes for an absolute path)
• SECRET_KEY: Flask secret key for session management
• DOCKER_DEFAULT_PLATFORM: Docker platform (e.g., linux/amd64, linux/arm64)

AI processing (Ollama):

• OLLAMA_URL: URL of the machine running ollama serve on the LAN, e.g. http://192.168.1.50:11434. From the dev compose on a Mac, use http://host.docker.internal:11434. The Ollama server must bind to 0.0.0.0 (set OLLAMA_HOST=0.0.0.0:11434 before ollama serve) for containers on other hosts to reach it.
• OLLAMA_VLM_MODEL: Vision model tag, e.g. qwen3.6:35b. Must be pulled on the Ollama host (ollama pull <tag>).
• OLLAMA_OCR_MODEL: Optional separate model for OCR runs. Leave unset to reuse OLLAMA_VLM_MODEL. Can also be set per profile in the UI.
• OLLAMA_NUM_CTX: Ollama context window (default 8192; Ollama's own default of 4096 overflows on image + prompt).
• OCR_PROMPT_DEFAULT / VLM_PROMPT_DEFAULT: Prompts used for each task. The prompt is recorded on each run so changes are auditable.
• PROCESSING_HOST: Identifier recorded on each run (default: hostname).
• WORKER_ENABLED: Toggle the background worker (default true).
• WORKER_POLL_INTERVAL_SEC, WORKER_TIMEOUT_SEC, WORKER_MAX_ATTEMPTS: Worker tuning knobs.

Volume Mappings

• <screenrecordings dir>:/app/storage: File storage directory
• ./data:/app/data: SQLite database (/app/data/files.db)

Health Checks

The Mac dev compose file (docker-compose.mbp.yaml) includes a health check that:

• Tests the application endpoint every 30 seconds
• Times out after 10 seconds
• Retries 3 times before marking unhealthy
• Has a 40-second start period

Development

Local Development

1. Create a virtual environment:

   python -m venv venv
   source venv/bin/activate  # Linux/Mac
   # or
   .\venv\Scripts\activate  # Windows
   

2. Install dependencies:

   pip install -r requirements.txt
   

3. Run the development server:

   flask run
   

4. Run the smoke test (uses a throwaway temp DB/storage, no setup needed):

   python test_smoke.py
   

Building for Different Platforms

To build for a specific platform (e.g., ARM64 for Synology):

DOCKER_DEFAULT_PLATFORM=linux/arm64 docker compose -f docker-compose.synology.yaml up -d

Security Considerations

There is no authentication — every endpoint, including destructive ones (POST /clear-database, DELETE /files/<id>), is open. Run this on a trusted LAN only. If it must be reachable from outside, put it behind a reverse proxy with basic auth and HTTPS.

AI Processing

How it works

1. New images are detected during a /scan and queued (one pending ProcessingRun per task: OCR + VLM). Existing images that are missing a completed run for either task are also re-queued automatically on each scan, so a scan doubles as a backfill.
2. A single background worker thread (started in-process inside the Flask container) polls for pending runs, claims one, base64-encodes the image, and POSTs to OLLAMA_URL/api/generate with images:[…].
3. On success the run is marked done, the output is stored, and the File.current_ocr_run_id / current_vlm_run_id pointer is updated so list queries stay one query.
4. If Ollama is unreachable (host asleep / wrong URL), runs stay pending without burning attempts and retry on the next poll.
5. Other failures retry up to WORKER_MAX_ATTEMPTS then mark the run failed (visible on the dashboard, can be retried with one click).

Concurrency is single-worker by design — Ollama serializes per model, and SQLite doesn't support SKIP LOCKED. If you ever want multiple workers across hosts, switch the queue to Postgres.

Setting up Ollama

On the machine that will run inference (Mac or CUDA workstation):

# Allow LAN connections
OLLAMA_HOST=0.0.0.0:11434 ollama serve

# Pull the model
ollama pull qwen3.6:35b

Verify reachability from the Flask container:

docker compose exec web python -c "import requests; print(requests.get('http://<ollama-host>:11434/api/tags').text)"

Endpoints

Dashboard:

• GET /processing — dashboard page
• GET /processing/status — JSON: counts by task/status, host/model breakdown, config (incl. paused state)

Job listing (paginated, filterable, sortable):

• GET /processing/runs?page=&per_page=&task=&status=&model=&file_search=&sort_by=&sort_order=
  • sort_by accepts: id, task, status, model, duration_ms, attempt, created_at, started_at, finished_at

Worker / queue control:

• POST /processing/pause — pause worker (state persisted)
• POST /processing/resume — resume worker
• POST /processing/retry-failed — reset all failed runs to pending
• POST /processing/cancel-pending — cancel every pending run (running runs untouched)
• POST /processing/clear — body { status?, task? } — delete runs (excludes running); omit fields to clear everything non-running
• POST /processing/reprocess-all — enqueue OCR+VLM for every image lacking pending/running runs

Per-run actions:

• POST /processing/runs/<id>/cancel — cancel a pending run
• POST /processing/runs/<id>/retry — reset a failed/cancelled run to pending

Per-file:

• GET /files/<id>/runs — full run history for a file
• POST /files/<id>/reprocess — body { "tasks": ["ocr","vlm"] }
• GET /files/<id>/metadata — now also returns ocr and vlm run snapshots

File listing filters/sort:

• GET /files?has_ocr=1&has_vlm=0 — filter by AI processing state
• GET /files?sort_by=ocr_updated|vlm_updated — sort by AI run finished_at
• GET /files?search=… — FTS5 full-text search (see Search section below)

Search

The file listing's search parameter is backed by a SQLite FTS5 virtual table (file_fts) that indexes:

• filename (the original filename)
• notes
• ocr_text (output of the file's current OCR run, if any)
• vlm_text (output of the file's current VLM/description run, if any)

The index is keyed by rowid = file.id and is kept in sync at well-defined points in app/fts.py:

• File ingestion / scan (/scan) — insert or update
• Scan reaper (file deleted from disk) — delete
• File metadata PUT (notes/tags) — update
• Worker on a done run — update with the new OCR or description text
• /clear-database — full rebuild

If the file_fts table is empty on startup but file has rows (e.g. you deployed this version on top of an existing DB), the index is automatically backfilled.

Query handling: each whitespace-separated token in the user input is escaped, wrapped in double quotes, and given an FTS5 prefix wildcard, then the tokens are ANDed (FTS5's default). For example, screen 2025 becomes "screen"* "2025"*. If the SQLite build lacks FTS5 (rare), search gracefully falls back to the legacy ILIKE match on filename only.

Performance Optimizations

• Pagination (20 items per page)
• Efficient database queries
• Proper indexing
• Caching where appropriate
• Optimized file serving
• Video streaming with range requests

Maintenance

Database Backup

The SQLite database lives in a dedicated bind-mounted data/ directory so it can be backed up independently of the source tree and the storage volume.

Inside the container the path is always /app/data/files.db (DATABASE_URL=sqlite:////app/data/files.db).

# Backup (use sqlite3 .backup for a consistent copy while running)
sqlite3 ./data/files.db ".backup './data/files.db.bak'"

# Or stop the container first, then a plain cp is safe:
docker compose stop && cp ./data/files.db ./data/files.db.bak && docker compose start

# Restore
docker compose stop && cp ./data/files.db.bak ./data/files.db && docker compose start

Migrating from a previous deployment (DB at instance/files.db):

docker compose down
mkdir -p data
mv instance/files.db data/files.db   # path may differ if you customised earlier
docker compose up -d --build

Logs

View container logs:

docker compose -f docker-compose.mbp.yaml logs -f

Updates

To update the application:

1. Pull latest changes:

   git pull
   

2. Rebuild and restart:

   docker compose -f docker-compose.mbp.yaml down
   docker compose -f docker-compose.mbp.yaml up -d --build
   

License

No license has been chosen yet (all rights reserved).