jvlinsta/receipt-video-extractor

Receipt Video Extractor

Turn a video of a stack of paper receipts into one clean, deduplicated image per receipt. Also dedupes against receipts the user already has, and uses a vision LLM to read merchant / date / total / category.

When to use this

The user has a video (phone clip) sweeping over several physical receipts — or flipping through a pile — and wants the individual receipts as separate, legible image files. Common context: expense reports, tax/bookkeeping, travel reimbursement. If they only have still photos, they can still use the dedup / enhancement pieces, but the main value is video → per-receipt images.

Prerequisites (one-time)

1. Python 3.10+. Install deps into a venv:

   cd <skill-dir> && python3 -m venv .venv && . .venv/bin/activate
   pip install -r requirements.txt
   

2. A vision model, one of:
   • Local (free): Ollama running a vision model, e.g. ollama pull qwen2.5vl:7b. Default.
   • Claude API (more accurate on hard thermal receipts): set ANTHROPIC_API_KEY, then pass --backend claude. pip install anthropic.

Core usage

. .venv/bin/activate
python scripts/extract_receipts.py <video.mp4> \
    --out ./out \
    [--existing <folder-of-receipts-you-already-have>] \
    --deskew --enhance \
    [--backend claude]        # default: ollama

Output: out/<date>_<merchant>_<total>.jpg for each unique new receipt, plus out/manifest.csv (merchant, date, total, currency, summary, category). Receipts that duplicate another frame — or anything in --existing — are dropped automatically. Existing-receipt signatures are cached per-model in <existing>/.receipt_signatures.<model>.json so the VLM reads each old receipt only once.

How it works (pipeline)

1. Keyframe extraction — a receipt held still is a low-motion plateau in the video; within each plateau it keeps the sharpest frame (variance of Laplacian). Falls back to interval sampling if there are no clear holds.
2. Pixel dedup — perceptual hash collapses near-identical frames.
3. Deskew (--deskew) — warps the largest quadrilateral flat.
4. Scan enhance (--enhance) — flattens lighting so paper goes white, local-contrast + unsharp so text pops (grayscale by default; --bw for hard black/white once crops are tight to the paper).
5. VLM extraction — reads {merchant, date, total, currency, summary, category}.
6. Semantic dedup — matches on total + month/day (year ignored — VLMs often misread the year) + fuzzy merchant, against other frames and --existing.

Useful flags

• --backend claude --claude-model claude-haiku-4-5 — cheaper Claude tier.
• --no-vlm — pixel dedup only, no VLM (fast, no OCR/semantic dedup).
• --dedup-existing — just report duplicate pairs within --existing, no video.
• --motion-frac 0.15 — lower = stricter about "held still".
• --min-hold 0.4 — min seconds a receipt is held to count.
• --phash-dist 6 — higher = more aggressive near-duplicate collapsing.

Standalone enhancement of existing images:

python scripts/enhance.py <folder-or-files> [--bw] [--upscale 1.5]

Filming tips (make it reliable)

Hold each receipt still ~1s on a plain, dark, matte surface filling most of the frame, then move to the next — the "hold → move → hold" rhythm is what the detector keys on. Even lighting; avoid glare on glossy thermal paper. Tight, full-frame framing lets --deskew crop cleanly and makes --bw usable.

Known limits / gotchas

• VLM misreads (esp. dates/merchant names on faded thermal paper) can let a true duplicate slip through as "new", or vice-versa. The manifest.csv is the human-verify step — eyeball it. Claude backend is markedly more accurate.
• Same-day, same-amount collisions (two €X purchases, same day, different shops) are separated only by merchant name — if the VLM reads it two ways, they may not merge. Don't loosen merchant matching below ~0.55.
• Very long receipts: film top→bottom slowly, or lay flat and shoot in sections.
• Motion blur from a fast pan is unrecoverable — enhancement improves contrast, not lost detail. Hold steady.

Privacy

Receipts are personal financial data. The local Ollama backend keeps everything on-device. The --backend claude path sends receipt images to the Anthropic API — use it when accuracy matters and that's acceptable.