Anionex/dsh-vision-toolkit

What does dsh-vision-toolkit do?

A DeepSeek Harness plugin that helps text-only models perform better on visual tasks: intent-aware image Q&A, long-screenshot OCR, UI restoration, grounding, pixel diff, Artifacts, and Web UI.

Features

• Intent-aware image Q&A: Ask questions about images; the model detects intent and provides relevant answers.
• Long-screenshot OCR: Extract text from long screenshots (e.g., chat history, web pages).
• UI restoration: Convert UI screenshots into code or markup (e.g., HTML, React components).
• Grounding: Map textual references to specific regions within an image.
• Pixel diff: Compare two images and highlight differences.
• Artifacts: Generate and manage visual artifacts.
• Web UI: Interactive interface for testing and debugging.

Usage

To use this toolkit with DeepSeek Harness, follow these steps:

1. Install the plugin in your DeepSeek Harness environment:

pip install dsh-vision-toolkit

2. Configure the tools in your agent's tool list:

tools:
  - name: image_qa
    description: "Answer questions about an image"
    input_schema:
      type: object
      properties:
        image_url:
          type: string
          description: "URL or base64 encoded image"
        question:
          type: string
          description: "The question about the image"
  - name: long_screenshot_ocr
    description: "Extract text from a long screenshot"
    input_schema:
      type: object
      properties:
        image_url:
          type: string
          description: "URL or base64 encoded screenshot"

3. Invoke the tools from your agent logic:

# Example: Image QA
response = agent.call_tool("image_qa", {
    "image_url": "https://example.com/screenshot.png",
    "question": "What is the title of this page?"
})
print(response)

Supported Platforms

• DeepSeek Harness (native)
• Compatible with agents running on Claude, Cursor, Codex, and any other platform that supports Harness.

License

MIT License - See LICENSE for details.

Contributing

Contributions are welcome! Please open an issue or submit a pull request on GitHub.