Scientific Data Visualization (research-figure-builder)
The goal is not "to draw a chart", but to ensure every data element in the figure can be traced back to its source data, and that others can reproduce the same deliverable using the attached snapshot and scripts.
When to Use
- Given scientific data tables and a figure creation task, you need real figures, statistical details, captions, and reproducibility evidence.
- You need comparisons, correlations, trends, distributions, compositions, category counts, or correlation heatmaps.
- You need to specify journal column width, canvas size, DPI, fonts, or export formats.
- You need to detect missing values, non-finite numbers, duplicates, small samples, and excessive categories, rather than silently removing data.
Not for experimental design, statistical method innovation, image post-processing beautification, or interactive dashboards. Significance markers are only accepted if they are provided along with the statistical method/correction details; asterisks are never auto-generated.
Input
-
Read the actual
/input/case.json,/input/task.md, and case materials; do not guess the task based on dataset name. -
Data supports inline
dataset.rows, CSV, TSV, JSON, and read-only SQLite. Relative paths are resolved relative to the case directory. When a case is moved to/app/workor similar directories, the system automatically falls back to look in ancestor directories'input/, the current directory, and/input(Harbor layout). If all lookups fail, report the full list of attempted paths; do not guess the data content. -
The figure request
figure_requests[]supports:figure_id,purpose,chart_hint/chart_type.x,y,group,variables,unit_id/replicate_unit,label.x_label,y_label,x_unit,unit/y_unit,title,panel.error_bar=sd|se|ci95|none,fit_group,significance.venue,width_mm,height_mm,dpi,format/formats.
-
The
data_dictionary'slabel/type/unit/missing_values/sentinelsfeed into labels, types, and cleaning rules. Numerical sentinels like-999must be explicitly declared; global guessing is not allowed. The shortlabelfrom the data dictionary takes precedence as the display name on axes (e.g., heatmaps); the original column name is retained in the lineage and analysis. -
If the case only has a natural language task and lacks
figure_requests, the Agent first generates a working case in/app/workthat preserves the originalcase_id, data references, andoutput_contract, only adding structured figure requests. Relative data paths do not need rewriting (fallback resolution per rule 2). Record this transformation in the evidence; do not fabricate data. Example request mapping (maintain the original task semantics; do not add conclusions not mentioned in the task):Task: "Compare expression differences among three genotypes, error bars with standard error" →
{"figure_id":"F1","purpose":"comparison","x":"Genotype","y":"Expression","error_bar":"se", "x_label":"Genotype","y_label":"Expression","title":"Comparison of expression levels among three genotypes"}The chart selection rationale is written into the request conversion record: purpose=comparison with explicit error bars →
bar_with_error_points(mean bar + all raw points); reject pie charts (unordered categories should not be encoded as angles).Task: "Association of petal length and width colored by species, with fit and confidence band" →
{"figure_id":"F2","purpose":"correlation","x":"petal_length_cm","y":"petal_width_cm", "group":"species","fit_group":true,"x_label":"Petal length","y_label":"Petal width"}Reject line plots (unordered observations should not be connected). Only group fits when
fit_group=true.Task: "Exploratory correlation structure of all variables" →
{"figure_id":"F3","purpose":"correlation","chart_hint":"heatmap"}(if variables are not listed, they are automatically identified from continuous columns in the data dictionary, and this decision is recorded).
Execution
Run in the production environment:
python3 -B /skills/weekly-skill/scripts/pipeline.py \
--case /input/case.json --task /input/task.md --out /app/output
python3 -B /skills/weekly-skill/scripts/validate.py \
--case /input/case.json --result /app/output/result.json
python3 -B /skills/weekly-skill/scripts/validate_figures.py \
--result /app/output/result.json
Dependencies: Rscript + ggplot2 + ggprism + svglite + jsonlite + systemfonts; Python auditing uses Pillow and pypdf. If dependencies are missing, save the snapshot, run plan, and scripts, mark the status as partial, and list the missing items. If the container lacks R packages but has network access, try a cold install and re-run:
Rscript -e 'install.packages(c("ggplot2","ggprism","svglite","jsonlite","systemfonts"), repos="https://cloud.r-project.org")'
python3 -m pip install --user Pillow pypdf
Installation failure should not block the process. Write dependency detection results to dependencies.json, and honestly degrade deliverables according to the partial completion criteria.
Pipeline
1. Freeze Input and Contract
Save source_data.tsv, the original source/hash, run_plan.json, cleaning rules, and dependency versions. Dynamically bind the case's output_contract. Any fields not produced are explicitly recorded in status.unmet.
2. Shell/Python: Data Preparation Only
data_prep.sh calls scripts/prepare_data.py to generate clean TSV files; no plotting:
- Default missing markers: empty strings, NA, N/A, NaN, null, None,
.. - Non-finite numbers and unparsable numeric values are excluded and their original row numbers and reasons are recorded.
- Fully duplicate rows are kept by default and reported; they are only deleted if
drop_duplicates=trueis explicitly set. - Categories are not limited to the top N; if there are many categories, keep all and adjust the layout.
- Output:
cleaned_data.tsv,figure_N.tsv,figure_N_lineage.json,data_quality.json.
3. R/ggplot2: Statistics and Plotting
plot.R reads only the TSV and run plan, calling theme.R:
| Route | Actual Expression |
|---|---|
bar_with_error_points | Raw jittered points + mean bar + explicit SD/SE/95% CI |
comparison_auto | n<3: points only; 3–9: points + declared summary; n≥10 per group: boxplot + jitter |
boxplot_points / violin_points | Explicit boxplot/violin + jitter; degrade to point plot when n<3 per group and record |
scatter_with_fit | Scatter + overall OLS with 95% confidence band; group fit only if fit_group=true; dual encoding with color + shape |
line_series | Sorted by true time/ordered numeric values, color + linetype; mean trend with repeated points, keep raw points |
histogram | Sturges binning, can be faceted by group |
stacked_bar | Explicit overall/group fields; tick marks set at cumulative totals; use sorted bars when no group |
category_count | Category count with sorted bars |
correlation_heatmap | Pairwise-complete Pearson; constant columns / insufficient valid pairs are NA; diverging color scale fixed to [-1,1] with colorbar |
SD/SE/CI for n=1, constant variable correlations, and zero X variance fits are all null, not written as 0. Bars start from 0; the highest tick mark covers the bar tops and error bars.
Automatic readability defenses (never truncating data): The base font size of the entire figure is driven by min(width, 1.55 × height); a wide canvas does not enlarge the font size. Scatter/line point sizes shrink by density based on "available plotting area / number of points" (minimum 0.9mm). Categories >8 switch to horizontal layout and automatically increase canvas height by row count (capped at max_height_mm). If row spacing is insufficient, category label font sizes shrink by row spacing (minimum 6pt). When vertical layout predicts label overlaps, rotate by 45°. Heatmap canvas height increases with variable count and width; axis labels use data dictionary short names preferentially and strip units in parentheses; rotation escalates from 45° to 90° when horizontal projection fails; font size shrinks by grid cell width; in-cell numbers are sized by grid cell width (maximum ~9pt, omitted when grid <5.5mm or variables >12). Time axis ticks are thinned according to canvas width and displayed as two-line labels "MM-DD / YYYY". Titles wrap by canvas width. Every adjustment is written into layout_adjustment, and into the analysis notes and warnings, making them traceable.
4. Three-Format Export
Each figure generates by default:
figure_N.pdf: Cairo PDF, actual page size and font embedding evidence are checked.figure_N.png: Default 600 dpi; the task can specify higher or lower but not below 300 dpi.figure_N.svg: svglite, retains editable<text>.figure_N.tsv+figure_N.csv,figure_N_spec.json,figure_N_analysis.json,figure_N_lineage.json.
Even if the user requests only PDF, PNG preview and SVG editable file are kept. Export format grading: if a requested format within the supported range (PDF/PNG/SVG) is missing, that figure must not be marked as complete. Unsupported requested formats (e.g., EMF) are recorded as format_gaps (out-of-scope gaps) and the default three formats are delivered; the format is not silently ignored or fabricated.
5. Four-Level QA
- Scientific Expression: Chart type, grouping, units, error bars, significance, and caption terminology are consistent.
- Numerical Verification: Python independently recalculates mean/SD/SE, OLS, Pearson, and correlation matrices, and compares with R results.
- File and Visual Inspection: easyplot checks size/DPI/PDF fonts/SVG text. Checks PNG for non-blank content, actual margins, and palette context. Margin semantics: blank figures or content touching/cropping (distance to canvas <0.02 in) are failures; whitespace below 0.1 in is only flagged as
tight, and a qualified figure is not demoted to partial. - End-to-End Reproducibility: Re-execute
data_prep.sh + plot.R + normalize_pdf.pyfrom the attached original snapshot, and compare SHA-256 of plotting TSV/CSV, lineage, statistics JSON, PDF, PNG, SVG. PDF normalization only sets Cairo/pypdf's CreationDate/ModDate to Unix epoch; page, font, or graphic content is never changed.
After automated checks complete, the Agent must visually inspect the actual PNG (and the PDF if necessary), checking for label overlap, garbled text, occlusion, and scientific semantics. Pixel margin checks are not equivalent to human/multimodal visual review.
Output
result.json preserves case_id and the original output contract, and includes:
figures: Per-figure route, actual chart type, preparation records, R analysis, independent captions, file metadata/hashes, QA, full-chain reproducibility results,layout_adjustment(layout adaptation records), andformat_gaps(out-of-scope format gaps).data_to_visual_map: Source column → visual channel, plotting TSV, and row-level lineage.baseline_comparison: Seven-dimensional determinism defense against baseline Agent/generic plotting solutions (chart selection/axes/encoding readability/annotation captions/export specs/reproducibility evidence/anomaly degradation). Each dimension notes typical failures, this skill's handling, and evidence location. Fair comparison at the platform level is run by the organizing entity; this table does not fabricate baseline numbers.evidence_map: Source snapshot, run plan, rule hashes, dependencies, scripts, and reproduction commands.status description: Gaps, explanation, and uncertainties consistent withstatus.overall.
All deliverable files are in /app/output. Can be independently re-run:
cd /app/output
bash data_prep.sh
Rscript plot.R
python3 -B scripts/normalize_pdf.py --plan run_plan.json --out .
python3 -B scripts/audit_outputs.py --plan run_plan.json --out .
Exception Handling
| Situation | Behavior |
|---|---|
| Case unparseable, duplicate column names, missing fields | Write honest error/partial result and gaps; do not fabricate figures |
| Missing/NaN/Inf/declared sentinels | Exclude unusable observations for that figure; keep source row number and reason |
| Duplicate rows | Keep by default and report; only deduplicate if explicit rule exists |
| A group has n=1 | Draw raw points; SD/SE/CI are null |
| Constant correlation / zero X variance | Draw only available points; correlation/fit are null |
| Too many categories or series | Do not truncate; auto-increase canvas height/horizontal layout/shrink label font size/facet, and record in layout_adjustment; when max height is exceeded, clearly indicate that labels may still be dense |
| Heatmap many variables / long labels | Canvas height increases with variable count and width; use dictionary short names + strip parenthetical units; angle adapts 45°→90°; font size shrinks by grid cell width (≥6pt); omit in-cell values when variables >12 |
| Dense date ticks | Thin ticks by canvas width; use two-line MM-DD/YYYY labels |
| Request for unsupported export format (e.g., EMF) | Record as format_gaps (out-of-scope) and deliver default three formats; do not mark as partial or silently ignore |
| Task provides significance results | Pass-through rendering: paired → bracket + annotation; single category → letters at group end; caption notes the method and says "no tests were calculated", never self-generate asterisks |
| Missing R package, font, Pillow, pypdf | Save re-runnable materials and explicitly list incomplete items; if network is available, install via cold-start command and re-run; when matplotlib is available, degrade to render a real PNG (300dpi+, PNG only; PDF/SVG gaps are honestly recorded, not fabricated) |
| Data file path failure | Fallback resolution across multiple base directories; if still failing, report all attempted paths |
| Single figure rendering failure | Continue with other figures; write status=error for that figure with the real reason |
| Any figure or QA failure | Overall status is partial; do not write complete |
Self-Check
# Run from the repository root
python3 -B answer/060_科研数据可视化/scripts/test_pipeline.py
python3 answer/_shared/verify.py --only 060
The minimal example must satisfy: general contract PASS, figure-specific PASS, all three formats exist for each figure, QA fully passed, and two full-chain runs have identical hashes. Only generated_at may change on repeated runs.
Method Sources and License
- Local
sci-plot: R/ggplot2, ggprism theme, Arial/Cairo, svglite, Sturges, tick mark coverage, and visual QA rules. - Rimagination/easyplot commit
d150e01: Reuseseasyplot_metadata.pyandeasyplot_palette_audit.py; MIT license text inscripts/easyplot-LICENSE.txt. - Chinese labels are a fidelity requirement for this input: default Arial; only when actual labels need CJK, select a real Chinese font and record it; do not treat glyph fallback as success.
Minimal Invocation Example
cd answer/060_科研数据可视化
python3 -B scripts/pipeline.py \
--case examples/min_example/input/case.json --out /tmp/figure060-demo
python3 -B scripts/validate.py \
--case examples/min_example/input/case.json --result /tmp/figure060-demo/result.json
python3 -B scripts/validate_figures.py --result /tmp/figure060-demo/result.json
The example uses 24 rows of synthetic data, containing one group comparison and one correlation plot. The example output is only for regression testing, not a formal case result.