DVC (Data Version Control)은(는) 무엇을 하나요?
설치
pip install dvc[s3]
pip install dvc[gs]
pip install dvc[azure]
pip install dvc[all]
cd my-ml-project
git init
dvc init
데이터 파일 추적
dvc add data/training_images/
dvc add data/dataset.csv
git add data/training_images.dvc data/dataset.csv.dvc .gitignore
git commit -m "DVC로 학습 데이터 추적"
원격 스토리지 설정
dvc remote add -d myremote s3://my-bucket/dvc-storage
dvc remote add -d myremote gs://my-bucket/dvc-storage
dvc remote add -d myremote /mnt/shared/dvc-storage
dvc push
dvc pull
재현 가능한 파이프라인 구축
stages:
prepare:
cmd: python src/prepare.py
deps:
- src/prepare.py
- data/raw/
outs:
- data/processed/
train:
cmd: python src/train.py
deps:
- src/train.py
- data/processed/
params:
- train.epochs
- train.learning_rate
- train.batch_size
outs:
- models/model.pkl
metrics:
- metrics/train.json:
cache: false
evaluate:
cmd: python src/evaluate.py
deps:
- src/evaluate.py
- models/model.pkl
- data/processed/
metrics:
- metrics/eval.json:
cache: false
plots:
- metrics/confusion_matrix.csv:
x: predicted
y: actual
train:
epochs: 50
learning_rate: 0.001
batch_size: 32
dvc repro
dvc repro train
실험 추적
dvc exp run --set-param train.learning_rate=0.01
dvc exp run --set-param train.learning_rate=0.001 --queue
dvc exp run --set-param train.learning_rate=0.01 --queue
dvc exp run --set-param train.learning_rate=0.1 --queue
dvc queue start --jobs 3
dvc exp show
dvc exp diff
dvc exp apply exp-abc123
dvc exp push origin exp-abc123
메트릭 및 플롯
import json
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, f1_score
import yaml
import pickle
with open("params.yaml") as f:
params = yaml.safe_load(f)["train"]
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = RandomForestClassifier(n_estimators=params["epochs"])
model.fit(X_train, y_train)
preds = model.predict(X_test)
metrics = {
"accuracy": accuracy_score(y_test, preds),
"f1_score": f1_score(y_test, preds, average="weighted"),
}
with open("metrics/train.json", "w") as f:
json.dump(metrics, f, indent=2)
with open("models/model.pkl", "wb") as f:
pickle.dump(model, f)
dvc metrics show
dvc metrics diff
dvc plots show metrics/confusion_matrix.csv
dvc plots diff
전체 클론 없이 데이터 접근
dvc get https://github.com/org/ml-repo data/processed/dataset.csv
dvc import https://github.com/org/ml-repo models/model.pkl
import dvc.api
with dvc.api.open("data/dataset.csv", repo="https://github.com/org/ml-repo") as f:
import pandas as pd
df = pd.read_csv(f)
url = dvc.api.get_url("models/model.pkl", repo="https://github.com/org/ml-repo")
주요 개념
.dvc 파일: 원격 스토리지의 대용량 데이터를 참조하는 작은 포인터 파일로, Git에 커밋됩니다.
dvc repro: 파이프라인 재현 — 변경된 종속성이 있는 단계만 재실행합니다.
- 실험(Experiments): 브랜치 없는 실험 추적 — 실행, 비교, 결과 적용.
- 매개변수(Params): 재현 가능한 구성을 위해 DVC가 추적하는 YAML 매개변수 파일.
- 메트릭(Metrics): 기본 제공 비교 도구가 있는 JSON/YAML 메트릭 파일.
- 원격 스토리지(Remote storage): S3, GCS, Azure, SSH, HDFS — 데이터는 원하는 위치에 저장됩니다.