macOS Vision Local OCR

This skill guides you in converting user-provided images/PDFs to text: using macOS's built-in Vision engine (macocr command) for local recognition, with no data uploaded.

Environment Check (Mandatory)

Before calling any OCR command, run the following check. If any condition is not met, stop and respond accordingly:

[ "$(uname -s)" = "Darwin" ] && echo OK-macOS
command -v macocr || ls "$HOME/.local/bin/macocr"

uname is not Darwin: This tool only supports macOS. Do not attempt to install or compile; directly inform the user that this Skill is only available on macOS and suggest using other OCR solutions.
macocr not found: Guide the user to run ./install.sh in this package directory (compilation requires Xcode Command Line Tools; on Apple Silicon without a compilation environment, it automatically falls back to the precompiled binary). After installation, run sh scripts/selftest.sh for an end-to-end self-test. Proceed only after it passes.

Once installation and self-test are complete, there is no need to repeat them for each session.

Standard Task Execution Flow

Follow these steps to handle the user's OCR request; do not skip steps:

Confirm input type. If the user provides an image (png/jpg/jpeg/heic/tiff), proceed directly to step 3. If it is a scanned PDF (no text layer), first go to step 2 to convert to images.
PDF to image conversion (only needed for scanned PDFs). Use the system's sips tool for rendering, starting at 144dpi:
```
sips -s format png -s dpiHeight 144 -s dpiWidth 144 input.pdf --out /tmp/p1.png
```
Note: sips handles only single pages; for multi-page PDFs, you must specify each page separately, or use pymupdf with a 4x matrix for rendering (significantly better for small-font course materials, see "How to Use Scanned PDFs").
Run recognition. The default command covers most scenarios:
```
macocr <image path>            # Multiple images can be listed sequentially; output separated by ===
```
Add parameters as appropriate per scenario (see parameter table below). The result is plain text, already arranged in visual line order.
Deliver results. Present the recognized text directly to the user; for table/number-dense results, remind the user to double-check critical numbers (see "Special Considerations for Table Data").

Parameter Quick Reference

Parameter	Purpose
`--fast`	Disable language correction. Recommended for tables and numbers to prevent numbers from being "corrected" to odd values
`--lang zh-Hans+en-US`	Default for mixed Chinese/English. Set to `en-US` for pure English for speed; use `zh-Hant` for Traditional Chinese
`--rect x,y,w,h`	Recognize only a rectangular region (origin top-left, pixels). Use when extracting only a specific table from a large image
`--upscale N`	For blurry small images (e.g., forwarded images <1000px wide) or old scans: first Lanczos upscale by N (1.0–4.0). Only try 2 → 3 incrementally when normal mode accuracy is poor; blindly upscaling sharp large images can cause stroke merging
`--orientation up\|left\|right\|180`	Manually correct camera rotation/flip. Vision does not auto-rotate based on EXIF; side-shot images must use this
`--list-langs`	List supported recognition languages on the machine (for installation self-test; output should include zh-Hans)

Troubleshooting Low Accuracy (in Order of Increasing Cost)

Try these from lowest to highest cost; do not jump to complex solutions immediately:

Get the original image: Forwarded chat images have very low recognition rate; first ask the user to provide the original file.
--rect to isolate the target region, reducing noise.
--upscale 2 (for blurry small images).
--fast (for table/number scenarios).
Image preprocessing (crop + grayscale + auto-contrast + sharpen + binarization), example below:

from PIL import Image, ImageOps, ImageEnhance
im = Image.open("src.JPG")
im.crop((x0, y0, x1, y1)).resize((w*2, h*2), Image.LANCZOS)   # crop + upscale
ImageOps.autocontrast(im.convert("L"))                         # grayscale + auto contrast
ImageEnhance.Sharpness(im).enhance(2.0)                        # sharpen
im.convert("L").point(lambda p: 255 if p > 128 else 0)         # binarize

Multi-version voting: Run 2–3 preprocessing variants on the same table, take the consensus parts, and separately review any inconsistent numbers. Cross-validation: If sums of whole rows/columns match, the result is likely reliable; if not, a number is wrong.

Why Does the Photos App's Live Text Seem More Accurate Than the Bare API?

It uses the same Vision engine. The difference comes from pre- and post-processing, not the model itself:

Orientation: "Photos" auto-rotates based on EXIF before recognition; the bare API defaults to .up → use --orientation to compensate.
Resolution: "Photos" uses the original full-resolution image; images in workflows are often compressed previews → always prefer the original file first.
Implicit downsampling: NSImage renders at "points"; @2x/@3x screenshots and some HEIC images lose half the pixels → This tool iterates over representations to get the largest bitmap.
Vertical text: Live Text handles vertical text; the bare API breaks it into fragmented lines → This tool auto-detects and applies confidence arbitration.
Interaction advantage (cannot be fully replicated): Live Text allows point selection, making errors feel "fewer"; CLI outputs the entire text at once, making every mistake visible.

Conclusion: Same engine, pipeline determines the ceiling.

Special Considerations for Table Data

OCR is most error-prone on tables: decimal points may be swallowed (. → ,), adjacent columns might merge into one string, and rows can be mixed up. Critical numbers must be double-checked: if a PDF original exists, extract its text layer for comparison; if only a photo is available, rely on multi-version voting plus row/column logic validation.

When Not to Use This Tool

Scenario	More Suitable Choice
The AI model itself can read images	Let it see the image directly; no need for OCR
Complex layout reconstruction needed (multi-column, formulas, table border structure)	Layout-aware parsing services like MinerU (data goes to cloud)
Batch converting entire scanned PDFs to text and cloud is acceptable	Cloud OCR service's batch interface

Rule: Local first, no data leakage; only go to cloud when higher accuracy is essential, and only after confirming data can be sent externally.

Verification & Troubleshooting

Post-installation self-test: sh scripts/selftest.sh (end-to-end test using package's test image; outputs PASS/FAIL).
Language list does not include zh-Hans: System language data is incomplete, very rare; check if macOS version ≥ 13.
Empty or completely wrong output: First confirm the image is readable (sips -g pixelWidth -g pixelHeight image), then try the "Troubleshooting Low Accuracy" steps incrementally.

System Requirements

macOS 13+ (automaticallyDetectsLanguage requires 13.0)
Xcode Command Line Tools for compilation (xcode-select --install); or use the precompiled Apple Silicon binary (scripts/macocr-arm64, ad-hoc signed).

Package File Reference

File	Purpose	When Needed
`install.sh`	Install macocr (compile first, precompiled fallback)	When environment check shows macocr is missing
`scripts/selftest.sh` + `scripts/selftest.png`	End-to-end self-test (script + built-in test image)	Verification after installation; run this first when troubleshooting recognition anomalies to confirm the tool itself works
`scripts/vision_ocr.swift`	Source code (~200 lines, auditable)	When needing to compile manually or audit
`scripts/macocr-arm64`	Precompiled Apple Silicon binary	For Apple Silicon machines without swiftc; automatically used by install.sh

GreenWich-art/agent-skills

What is agent-skills?

Ask in your favorite AI

Documentation