Communitygithub.com

DVC (Data Version Control)

ML 프로젝트를 위한 데이터 버전 관리. 대규모 데이터셋과 모델을 Git과 함께 추적하고, 재현 가능한 ML 파이프라인을 구축하며, 메트릭 비교로 실험을 실행합니다. S3, GCS, Azure 및 로컬 파일 시스템을 포함한 모든 스토리지 백엔드와 호환됩니다.

DVC (Data Version Control)란 무엇인가요?

이 스킬은 ML 프로젝트를 위한 데이터 버전 관리 도구인 DVC(Data Version Control)의 포괄적인 사용법을 제공합니다. DVC는 Git과 유사한 방식으로 대용량 데이터셋, 모델, 메트릭을 추적할 수 있게 해주며, Git 자체는 대용량 파일을 효율적으로 처리하지 못하는 문제를 해결합니다. 사용자는 `dvc add` 명령어로 데이터 파일을 추적하고, `.dvc` 포인터 파일을 Git에 커밋하며, `dvc repro`로 변경된 단계만 재실행하여 재현 가능한 ML 파이프라인을 구축할 수 있습니다. 또한, 파라미터 및 메트릭 추적을 지원하며, `dvc exp` 명령어를 통해 실험을 실행, 비교, 적용할 수 있습니다. S3, GCS, Azure Blob Storage, SSH, HDFS 등 다양한 원격 스토리지를 지원하여 데이터가 원하는 위치에 저장되도록 보장합니다. 이 스킬은 Claude Code, Cursor와 같은 터미널 기반 AI 에이전트와 호환되며, DVC의 전체 워크플로우를 안내하는 데 유용합니다.

지원 대상~Claude Code~Codex CLI~Cursor
npx skills add https://github.com/TerminalSkills/skills/tree/HEAD/skills/dvc

즐겨 사용하는 AI에게 물어보기

이 에이전트 스킬이 미리 로드된 새 채팅을 엽니다.

문서

DVC (Data Version Control)은(는) 무엇을 하나요?

설치

# 클라우드 스토리지 지원과 함께 DVC 설치
pip install dvc[s3]    # AWS S3 용
pip install dvc[gs]    # Google Cloud Storage 용
pip install dvc[azure] # Azure Blob Storage 용
pip install dvc[all]   # 모든 리모트 지원

# Git 저장소에서 DVC 초기화
cd my-ml-project
git init
dvc init

데이터 파일 추적

# track_data.sh — 대용량 파일을 Git 대신 DVC 추적에 추가
# 대용량 데이터셋 추가
dvc add data/training_images/
dvc add data/dataset.csv

# DVC가 .dvc 파일(작은 포인터) 생성 — 이것들을 Git에 커밋
git add data/training_images.dvc data/dataset.csv.dvc .gitignore
git commit -m "DVC로 학습 데이터 추적"

원격 스토리지 설정

# setup_remote.sh — DVC가 실제 파일 내용을 저장할 위치 구성
# S3
dvc remote add -d myremote s3://my-bucket/dvc-storage

# Google Cloud Storage
dvc remote add -d myremote gs://my-bucket/dvc-storage

# 로컬 / 네트워크 경로
dvc remote add -d myremote /mnt/shared/dvc-storage

# 데이터를 원격으로 푸시
dvc push

# 원격에서 데이터 가져오기 (다른 머신이나 클론 후)
dvc pull

재현 가능한 파이프라인 구축

# dvc.yaml — 종속성 및 출력이 있는 ML 파이프라인 단계 정의
stages:
  prepare:
    cmd: python src/prepare.py
    deps:
      - src/prepare.py
      - data/raw/
    outs:
      - data/processed/

  train:
    cmd: python src/train.py
    deps:
      - src/train.py
      - data/processed/
    params:
      - train.epochs
      - train.learning_rate
      - train.batch_size
    outs:
      - models/model.pkl
    metrics:
      - metrics/train.json:
          cache: false

  evaluate:
    cmd: python src/evaluate.py
    deps:
      - src/evaluate.py
      - models/model.pkl
      - data/processed/
    metrics:
      - metrics/eval.json:
          cache: false
    plots:
      - metrics/confusion_matrix.csv:
          x: predicted
          y: actual
# params.yaml — 파이프라인 매개변수 (DVC로 추적)
train:
  epochs: 50
  learning_rate: 0.001
  batch_size: 32
# 전체 파이프라인 실행 (변경된 단계만 재실행)
dvc repro

# 특정 단계 실행
dvc repro train

실험 추적

# experiments.sh — ML 실험 실행 및 비교
# 수정된 매개변수로 실험 실행
dvc exp run --set-param train.learning_rate=0.01

# 여러 실험을 병렬로 실행
dvc exp run --set-param train.learning_rate=0.001 --queue
dvc exp run --set-param train.learning_rate=0.01 --queue
dvc exp run --set-param train.learning_rate=0.1 --queue
dvc queue start --jobs 3

# 실험 비교
dvc exp show
dvc exp diff

# 성공적인 실험을 작업 공간에 적용
dvc exp apply exp-abc123

# 실험을 Git 브랜치로 푸시
dvc exp push origin exp-abc123

메트릭 및 플롯

# train.py — DVC 추적 메트릭을 출력하는 학습 스크립트
import json
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, f1_score
import yaml
import pickle

# 매개변수 로드
with open("params.yaml") as f:
    params = yaml.safe_load(f)["train"]

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = RandomForestClassifier(n_estimators=params["epochs"])
model.fit(X_train, y_train)

preds = model.predict(X_test)
metrics = {
    "accuracy": accuracy_score(y_test, preds),
    "f1_score": f1_score(y_test, preds, average="weighted"),
}

with open("metrics/train.json", "w") as f:
    json.dump(metrics, f, indent=2)

with open("models/model.pkl", "wb") as f:
    pickle.dump(model, f)
# 실험 간 메트릭 보기
dvc metrics show
dvc metrics diff

# 플롯 생성
dvc plots show metrics/confusion_matrix.csv
dvc plots diff  # 실험 간 플롯 비교

전체 클론 없이 데이터 접근

# 전체 클론 없이 모든 DVC 저장소에서 추적 파일에 접근
dvc get https://github.com/org/ml-repo data/processed/dataset.csv
dvc import https://github.com/org/ml-repo models/model.pkl
# dvc_api.py — 프로그래밍 방식으로 DVC 추적 파일에 접근
import dvc.api

# DVC 저장소에서 파일 읽기
with dvc.api.open("data/dataset.csv", repo="https://github.com/org/ml-repo") as f:
    import pandas as pd
    df = pd.read_csv(f)

# 추적된 파일의 URL 가져오기
url = dvc.api.get_url("models/model.pkl", repo="https://github.com/org/ml-repo")

주요 개념

  • .dvc 파일: 원격 스토리지의 대용량 데이터를 참조하는 작은 포인터 파일로, Git에 커밋됩니다.
  • dvc repro: 파이프라인 재현 — 변경된 종속성이 있는 단계만 재실행합니다.
  • 실험(Experiments): 브랜치 없는 실험 추적 — 실행, 비교, 결과 적용.
  • 매개변수(Params): 재현 가능한 구성을 위해 DVC가 추적하는 YAML 매개변수 파일.
  • 메트릭(Metrics): 기본 제공 비교 도구가 있는 JSON/YAML 메트릭 파일.
  • 원격 스토리지(Remote storage): S3, GCS, Azure, SSH, HDFS — 데이터는 원하는 위치에 저장됩니다.

Individual skills in this repo

This repo contains 20 individual skills — each has its own dedicated page.

TerminalSkills/skills

>- Run GitHub Actions locally with act. Use when a user asks to test GitHub Actions workflows locally, debug CI pipelines without pushing, or run workflows offline.

TerminalSkills/skills

>- You are an expert in AG2 (formerly AutoGen), the open-source multi-agent conversation framework. You help developers build systems where multiple AI agents collaborate through structured conversations — with tool use, human-in-the-loop, code execution, group chat orchestration, and nested conversations — for complex tasks like software development, research, and data analysis.

TerminalSkills/skills

>- Generate short-form videos with AI — script writing, text-to-speech narration, stock footage selection, subtitle generation, and video assembly. Use when: creating TikTok/YouTube Shorts/Reels content, automating video production, building content pipelines.

TerminalSkills/skills

>- Animate 3D objects and characters in Blender with Python. Use when the user wants to keyframe properties, create armatures and rigs, set up IK/FK chains, animate shape keys for facial animation, edit F-Curves, use the NLA editor to blend actions, add drivers for expression-based animation, or script any animation workflow in Blender.

TerminalSkills/skills

>- Automate motion capture and tracking workflows in Blender with Python. Use when the user wants to import BVH or FBX mocap data, retarget motion to armatures, track camera or object motion from video, solve camera motion, clean up motion capture data, or script any tracking pipeline in Blender.

TerminalSkills/skills

>- You are an expert in Bolt.new by StackBlitz, the AI-powered full-stack development environment that runs entirely in the browser. You help developers go from idea to deployed app in minutes using natural language prompts — Bolt generates complete applications with frontend, backend, database, and deployment, all running in a WebContainer without local setup.

TerminalSkills/skills

>- Assists with using Bun as an all-in-one JavaScript/TypeScript runtime, package manager, bundler, and test runner. Use when building HTTP servers, managing packages, running tests, or migrating from Node.js. Trigger words: bun, bun serve, bun install, bun test, bun build, javascript runtime, bun runtime.

TerminalSkills/skills

>- Self-host customer support with Chatwoot. Use when a user asks to set up open-source customer support, add live chat without SaaS costs, build a multi-channel inbox, or deploy a free Intercom alternative.

TerminalSkills/skills

>- You are an expert in Chi, the lightweight, idiomatic Go HTTP router built on `net/http`. You help developers build composable HTTP services using Chi's middleware stack, route groups, URL parameters, sub-routers, and context-based request scoping — providing Express-like ergonomics while staying 100% compatible with Go's standard library.

TerminalSkills/skills

When the user wants to edit, review, or improve existing marketing copy. Also use when the user mentions 'edit this copy,' 'review my copy,' 'copy feedback,' 'proofread,' 'polish this,' 'make this better,' or 'copy sweep.' This skill provides a systematic approach to editing marketing copy through multiple focused passes.

TerminalSkills/skills

>- Add live chat and customer support with Crisp. Use when a user asks to add a chat widget, implement live customer support, set up a help desk, create a knowledge base, or add a chatbot to a website.

TerminalSkills/skills

>- Assists with building custom interactive data visualizations using D3.js. Use when creating charts, graphs, maps, force layouts, or hierarchical diagrams that require fine-grained control beyond what charting libraries provide. Trigger words: d3, data visualization, chart, svg, scales, force graph, treemap, choropleth.

TerminalSkills/skills

dbt (data build tool) transforms data in your warehouse using SQL SELECT statements. Learn project setup, models, tests, documentation, incremental materializations, and integration with data warehouses like PostgreSQL, BigQuery, and Snowflake.

TerminalSkills/skills

>- You are an expert in dlt, the open-source Python library for building data pipelines. You help developers load data from any API, file, or database into warehouses and lakes using simple Python decorators — with automatic schema inference, incremental loading, and built-in data contracts. dlt is the "requests library for data pipelines.

TerminalSkills/skills

>- You are an expert in E2B, the cloud platform for running AI-generated code in secure sandboxes. You help developers give AI agents the ability to execute code, install packages, read/write files, and run long processes in isolated cloud environments — each sandbox is a lightweight VM that boots in ~150ms with full Linux, filesystem, and networking.

TerminalSkills/skills

>- Transcode, convert, edit, and process audio and video with FFmpeg. Use when a user asks to convert video formats (webm to mp4, mkv to mp4), extract audio from video, compress video files, trim or cut clips, concatenate videos, add subtitles, create thumbnails, apply filters, change resolution or bitrate, re-encode media, create GIFs from video, add watermarks, normalize audio, stream media, or build automated media processing pipelines.

TerminalSkills/skills

>- Create advanced animations with GSAP. Use when building complex timelines, scroll-driven animations, SVG morphing, text animations, or high-performance web animations beyond CSS capabilities.

TerminalSkills/skills

>- Generate deterministic MP4 videos from HTML, CSS, media, and seekable animations using HeyGen's HyperFrames framework. Use when someone asks to "render HTML to video", "make a video with HyperFrames", "create a programmatic video", "turn an animation into MP4", "build a launch/product video from HTML", "render GSAP/Lottie/Three.js to video", or set up an agent-driven video pipeline. Covers init/preview/render/add/lint/inspect commands, data-* timing attributes, animation adapters, and the component catalog.

TerminalSkills/skills

When the user wants to perform load testing, stress testing, or performance testing of APIs and websites using k6. Also use when the user mentions "k6," "load test," "performance test," "stress test," "spike test," "soak test," "thresholds," "virtual users," or "VUs." For browser-based testing, see selenium.

TerminalSkills/skills

>- Lovable (formerly GPT Engineer) is a browser-based AI app builder that turns a natural-language description into a working full-stack web app with a built-in backend, authentication, database and one-click publishing. Use when the user wants to prototype or ship a web app with Lovable, write effective Lovable prompts, connect Supabase or GitHub, review generated security rules, or move a Lovable project into their own editor.

관련 스킬