alt — Auto-Caption
Runs a local VLM over each image in input_dir and writes a JSON sidecar to
output_dir. Useful for auto-generating captions, understanding batch content,
or feeding descriptions into other tools.
The skill directory (where this SKILL.md lives) is referred to as $SKILL_DIR below.
When to Use
Use this skill when the user wants to:
- Auto-generate Instagram captions from photos
- Understand what's in a batch of images without reviewing them manually
- Feed image descriptions into a content pipeline or scheduling tool
Setup (first run only)
cd "$SKILL_DIR" && uv sync
moondream2 weights are downloaded on first use (~4 GB from HuggingFace). Phi-4 weights are larger (~8 GB) and require a GPU.
Agent Workflow
1. Ask the user
Before I describe the images, I need to know:
🧠 Model
- moondream2 — 2B params, fast, works on CPU (default)
- phi4 — 3.8B multimodal, higher quality, requires GPU
💬 Caption style (optional — I'll use a sensible default)
e.g. "Casual and fun with emojis" or "Professional and inspiring"
⚙️ Device
- auto — GPU if available, else CPU (default)
- cpu — moondream2 only; ~7s/image on a modern desktop CPU
📁 Input / output directories (default: ./input and ./output)
2. Edit config.json
If the user gives a caption style, incorporate it into prompt_caption.
Example: "Write a fun, emoji-rich Instagram caption for this image. Include 3-5 hashtags."
If the user asks for branded visuals, check skills/brand-manager/brand-assets/asset-manifest.json for available fonts and mention the preferred brand font in the generated caption guidance or handoff notes. This skill writes JSON sidecars only, so it does not render text itself; downstream renderers such as visual-hook, video-captioner, and end-cta should use the actual brand font files when they are specified and available.
3. Run
cd "$SKILL_DIR" && uv run python scripts/describe.py --config config.json
The first run downloads model weights — warn the user this may take a few minutes.
4. Report results
Show the description and caption for each image. Tell the user where the JSON files were written.
Config Reference
| Key | Values | Default | Description |
|---|---|---|---|
input_dir | path | ./input | Source image folder |
output_dir | path | ./output | Destination folder for JSON sidecars |
model | moondream2, phi4 | moondream2 | Vision-language model |
prompt_description | string | "Describe this image in one sentence." | Prompt for the description field |
prompt_caption | string | "Write an engaging Instagram caption…" | Prompt for the caption field |
device | auto, cpu, cuda | auto | Inference device |
Common Invocations
# Default (moondream2, auto device)
cd "$SKILL_DIR" && uv run python scripts/describe.py
# Higher quality (GPU required)
cd "$SKILL_DIR" && uv run python scripts/describe.py --model phi4
# Force CPU
cd "$SKILL_DIR" && uv run python scripts/describe.py --device cpu
# Custom caption prompt
cd "$SKILL_DIR" && uv run python scripts/describe.py \
--prompt-caption "Write a professional LinkedIn caption for this image."
Output
For each photo.jpg in input_dir, writes photo.json in output_dir:
{
"description": "A woman in a sunlit café holding a coffee cup.",
"caption": "Morning rituals ☕✨ Starting the day right. #coffeetime #morningvibes #cafe",
"tags": ["portrait", "coffee", "indoor", "warm light"]
}
When brand fonts are relevant, include the font name/tag in downstream instructions rather than embedding a font file in this JSON. Rendering skills are responsible for resolving the manifest path and applying the font.
Error Handling
phi4+cpu→ rejected at config validation with a clear message- Model download failures → shown with HuggingFace error details
- Individual image errors → logged; other images continue
Remote Inference
This skill supports optional remote captioning while preserving the same JSON sidecar output.
- Default behavior is still local
- Remote providers are opt-in only
- Supported providers:
huggingface,replicate
Config keys
| Key | Default | Notes |
|---|---|---|
provider | null | null, local, or none keeps local mode |
remote_model | null | Optional provider-specific VLM model override |
hf_token_env | HF_TOKEN | HuggingFace auth env var name |
replicate_api_key_env | REPLICATE_API_TOKEN | Replicate auth env var name |
remote_timeout_seconds | 300 | Remote call timeout |
Examples
# HuggingFace remote image captioning
export HF_TOKEN=hf_your_token
uv run python scripts/describe.py --config config.json --provider huggingface
# Replicate remote image captioning
export REPLICATE_API_TOKEN=r8_your_token
uv run python scripts/describe.py --config config.json --provider replicate --remote-model <replicate-model-slug>
Notes
- Remote mode still writes the same
description/caption/tagsJSON sidecar - Missing credentials fail fast; there is no silent fallback to local mode
- Replicate requires an explicit model slug for remote captioning
Caption Quality Standards
The caption is as important as the image. A weak caption wastes a strong visual. Apply these principles when generating or prompting for captions.
Rule of One
Every caption should carry one clear central idea. If the caption tries to make multiple points, it becomes diffuse and loses impact. The strongest captions have a single spear tip: one promise, one lesson, one observation, one story beat.
Test: can you say this caption in one sentence? Do all the lines support the same idea? If not, cut until the core is clear.
Hook first
The first line of any caption is the hook. On Instagram, only the first ~125 characters are shown before "more" — that first line must earn the tap. On LinkedIn, the first line is the headline.
Hook patterns that work:
- Contrarian: "Stop doing X."
- Curiosity gap: "Most people don't realize..."
- Identity callout: "If you're [X], pay attention."
- Authority/experience: "If I had to start over, here's what I'd do."
- Pain-point: "This is why you're stuck at [X]."
Avoid opening with "I" or with a warm-up sentence that says nothing. The first line should earn the rest of the read.
Platform-specific structure
Instagram:
- Hook first (≤125 chars)
- Body: story, lesson, or insight
- 3–5 relevant hashtags at the end
- Authentic tone — avoid polished marketing language
- Emoji are acceptable but should reinforce meaning, not decorate
LinkedIn:
- 600–800 characters for best reach; 100–150 for high CTR when needed
- Strong hook in the first line
- No external links in the post body (LinkedIn suppresses reach for outbound links)
- Frameworks that work: Hook → insight → breakdown → CTA; or Story → lesson → action → engagement question
- Clarity over creativity — readers scan quickly
- Avoid: emoji overload, excessive whitespace, back-to-back same format, external fonts
Anti-generic test
Before accepting a generated caption, ask: does this sound like the expert who took this photo, or does it sound like generic AI writing? If it could have been written by anyone about anything, it fails. The caption must reflect a specific point of view, observation, or lived moment.
Avoid
- Opening with "I" as the first word
- Vague inspirational statements without a concrete point
- Emoji used as visual padding rather than to reinforce meaning
- More than one call-to-action in a single caption
- Hashtag spam (10+ hashtags) on LinkedIn