What does Empirical Research Pipeline do?
Turn "a pile of messy materials" into a verifiable sample flow, traceable coded/extracted data, reproducible models, and submission-ready charts and manuscripts. This skill is not limited by discipline, language, or file format; legal court rulings are just one example of "text coding" — questionnaires, interviews, news articles, policy documents, experimental records, open data, web pages, and database exports are all applicable.
The example numbers, filenames, paths, and variable values in this repository/package are fictional placeholders and do not contain any real data, coded data, SPSS data, or complete manuscripts.
When to Use
- The user provides a folder containing a mix of PDFs, Word files, Excel files, CSVs, text files, web pages, interview transcripts, images, or scanned documents and requires organization, coding, statistics, charts, or a written paper.
- Existing CSV/SPSS data needs cleaning, merging, appending new materials, adjusting sample criteria, or re-running models.
- Questionnaire, interview, experiment, observation, or public data require descriptive statistics, between-group comparisons, regression, heterogeneity, robustness checks, or predictive evaluation.
- A Word/PDF output is needed, and the numbers in figures, tables, and the body text must be consistent.
- The user says "I need to do the same thing again with similar materials" and wants to reuse the same workflow.
Material Types and Processing Boundaries
| Material | Native Processing | Notes |
|---|---|---|
| Text layer, page count, character count, image count, encryption status | Scans require OCR first; image-only PDFs are flagged as image_only | |
| DOCX / ODT / PPTX | Paragraphs, tables, slide text | Old .doc/.ppt formats need conversion first |
| XLSX / XLSM | Worksheets, cell text; read-only open | .xls needs conversion first; install openpyxl recommended |
| CSV / TSV / JSON / JSONL | Fields, row count, encoding, missing values; tabular data quality audit | Coding worksheets skip tabular data and proceed to auto_stats |
| TXT / MD / HTML / XML / YAML / RTF | Plain text extraction; HTML de-tagging | Web page content is usable; scripts/styles are removed |
| Images (PNG/JPG/TIFF, etc.) | File listing; dimensions/page count | OCR first, then enter coding; this skill does not include OCR |
| Audio/Video | File listing | Transcribe/extract frames first, then enter coding; this skill does not include ASR |
| Archives | File listing | Unzip first; audit flags as unsupported |
| Unknown/Proprietary Formats | File listing | Convert or export to one of the above formats first |
"Any material" does not mean "any binary can be directly parsed." This skill first includes all files in the audit and sample reconciliation; formats that cannot be read directly are clearly flagged with conversion instructions and are not silently omitted.
Six Hard Rules
- Audit first, then code/model. Do not start statistics before you know the file count, number of readable texts, duplicates, field missingness, and reasons for exclusion.
- Deduplicate by content, not by filename. Same content with different names and different content with the same name are common; use MD5 at minimum, then verify with title/key fields/body sampling.
- One row = one analysis unit. Documents, respondents, cases, organizations, region-years, or questionnaire responses must be defined and fixed at project start. Explain in limitations if multiple units are mixed.
- Every code/extracted value must have evidence. If the original material does not contain the information, record
NA/not_recorded; do not default to 0. Negative assertions, others' suggestions, or paraphrases cannot be treated as ultimate facts. - Describe sample N ≠ Model sample N. Report N and selection criteria separately for each model; use the same criteria for body text, tables, figures, and abstract.
- Association ≠ Causation. Use "association/correspondence/more likely," not "causes/leads to/proves effective."
Workflow Overview
0 Scope & Question → 1 Material/Data Audit → 2 Coding/Extraction Protocol → 3 Dataset
↓
7 Final Check ← 6 Manuscript Assembly ← 5 Publication-grade Charts ← 4 Statistical Analysis
Each stage has a "gate." If the gate is not passed, do not proceed to the next stage.
Stage 0: Scope & Question
First, write a one-page scope.md that includes at least:
- Research question, theoretical mechanism, expected contribution.
- Material types and sources: database, search strategy/sampling frame, time range, language, region.
- Unit of analysis: one document/one respondent/one case/one organization/one questionnaire.
- Inclusion/exclusion criteria: time, type, language, length, duplicates, unreadable files, missing key fields.
- Expected variable list and roles: dependent variable, key independent variable, mechanism/moderating variable, control variables.
- Data form: text, tabular, mixed; whether manual coding, questionnaire scales, experimental groups, or secondary data is needed.
Gate: Be able to explain in one sentence how many cases are lost at each step from "raw materials → readable materials → in-scope sample → analysis sample," and why.
Stage 1: Material & Data Audit (Must Be Run First)
1A File-Level Audit
Run scripts/audit_corpus.py:
python scripts/audit_corpus.py --root "data/materials" --out-dir "output/audit" --min-chars 100
Default scans all supported types; no need to write individual --ext flags; use --list-ext to see the full list. It outputs:
| File | Purpose |
|---|---|
corpus_audit.csv | For each file: extension, type, bytes, pages/tables, characters, images, MD5, status, error |
corpus_summary.json | Machine-readable summary |
corpus_duplicates.csv | Duplicate groups, keeper/removal suggestions |
corpus_report.md | Readable report that can be pasted into project logs |
State machine (every file must fall into one category):
| Status | Meaning | Enters Analysis? |
|---|---|---|
valid | Can be opened, has sufficient text/data | Yes |
metadata | ._*, .DS_Store, etc. | No |
hidden | Hidden files/directories | No, unless confirmed by user |
unsupported | Unknown format, archive, old Office, missing parser | No, convert/unzip first |
quarantined | Located in _duplicates or similar skip directories | No, isolated copies |
zero_byte | 0 bytes | No |
corrupt | Cannot open, file damaged | No |
encrypted | Password/restricted access | No, unless decrypted |
textless | Has pages/cells but insufficient text | No, check if empty or needs re-export |
image_only | Image/scan, no text layer | Usable after OCR |
media | Audio/video | Transcribe/extract frames first |
Duplicates are not a separate status: content-identical files are still valid, but are marked with dup_group/dup_role as keeper/duplicate; the report's unique_valid_files already excludes copies.
Handle each case:
._*,.DS_Store: macOS metadata, not research materials; count as "non-material files," not in the sample. Readable._*files are noted in the error column; requires manual confirmation.- Hidden files/directories: mark as
hidden; extensions not in the support list:unsupported; skip directories like_duplicates:quarantined. None enter the sample, but all must appear in totals and the report. - Zero bytes, unopenable, encrypted, insufficient text: record as
zero_byte/corrupt/encrypted/textless; image-only PDFs:image_only. - Top-level/year distribution: count only unique samples (
validand not duplicate); do not include metadata, corrupt files, or duplicates in the sample structure. - Duplicate copies: group by MD5; keep the version with identifiers/year, shorter path, and complete text; move the rest to
_duplicates/; do not delete directly. - Truncated filenames, trailing dots, overly long paths: common on Windows; read with Python and record in the report; do not count manually via File Explorer.
- Same name, different content; same body text but different metadata/export settings (different MD5): must be manually verified by title/key fields/body sampling; do not rely solely on MD5 for deduplication.
Then run scripts/dedupe_corpus.py to generate a deduplication plan:
python scripts/dedupe_corpus.py --audit "output/audit/corpus_audit.csv" --root "data/materials" --out-dir "output/audit" --apply
Gate: sample_flow.md must reconcile:
Raw file count (including .DS_Store/files without extensions)
- Metadata/hidden files
- Unsupported/needs conversion (unsupported)
- _duplicates and similar isolated directories (quarantined)
- Zero byte/corrupt/encrypted/insufficient text
= Readable material count
- Out of scope (time/type/language)
= Valid material count
- Duplicate copies
= Unique sample count (based on unique_valid_files)
- Missing key variables/unable to code
= Analysis sample count
Each model also has its own N; do not rely on a single N for everything.
1B Tabular Data Quality Audit
If the materials are CSV/Excel/SPSS data, perform field-level checks before coding/modeling:
- Row count, column count, type per column (numeric/categorical/text/date), missing rate, number of unique values.
- Is the primary key/ID unique? Duplicate rows? Multiple records for the same unit?
- Are categorical variable values inconsistently spelled ("Yes/No/1/0/TRUE" mixed)? Date range.
- Continuous variable range, outliers, P99 truncation, whether log transformation is needed.
- Questionnaire scale reverse-coded items, split multiple-choice options, unit consistency.
- Encoding (UTF-8-SIG/GB18030), decimal separator, SPSS variable name legality.
csv_to_spss_syntax.py cleans illegal/duplicate variable names and generates .sps; auto_stats.py provides descriptive statistics, between-group tests, and correlations to find data issues, but does not replace formal modeling.
Gate: Be able to explain the source, type, missingness handling, outlier handling, and final model inclusion for every column.
Stage 2: Coding/Extraction Protocol
2.1 Text Coding
- First, create an instance of
assets/codebook-template.md: each variable must have a definition, values, decision rules, evidence location, positive example, negative example, and missing rule. - Use
scripts/make_coding_sheet.pyto extract full text and candidate evidence sentences, generating a coding worksheet; it only provides evidence segments, not final judgments.- Default
--profile generic: suitable for interviews, news, policy, reports, web pages, meeting minutes, etc. --profile legal: additionally extracts court, case number, province, and cause of action candidates; use only when processing Chinese court judgments.- Tabular data (CSV/TSV/JSON/XLSX) is skipped with a prompt to use the statistical workflow; images/audio/video are prompted for OCR/transcription first.
- Default
- During manual/LLM coding, each record must save at least one
*_evevidence sentence; a code without an evidence sentence is considered incomplete. - Have 10–15% of materials double-coded by a second coder or in a second round; report agreement rate or Cohen's κ; resolve disagreements before revising the codebook.
- Freeze the codebook before full coding; later changes must record the version and scope of impact.
2.2 Questionnaire/Scale/Structured Extraction
- First, create a project-level variable dictionary: item number, item text, values, reverse-coded items, missing codes.
- Split multiple-choice/ranking items into 0/1 columns or multiple columns; check scale reliability (α/ω) and dimensionality before calculating scale totals.
- Open-ended items can be thematically coded first, then transformed into count/categorical variables; do not treat word frequencies as themes.
- Primary keys, unit identifiers, and time fields in tabular data must be traceable back to the original file.
Gate: Randomly sample 10 records; must be able to trace from code value → evidence sentence/original field → original text/row; otherwise, return to coding table to fix rules.
See references/02-coding-protocol.md for details.
Stage 3: Dataset
Recommended three files:
data/coded.jsonl: coding details for each material + evidence snippets, for traceability.data/analysis.csv: analysis-wide table, one row per analysis unit.data/analysis.sav: openable directly in SPSS, with variable labels and value labels.
Required marker columns:
| Column | Meaning |
|---|---|
in_scope | Whether it falls within the research scope (time, type, region, language) |
in_analysis | Whether it enters descriptive statistics (excluded if uncodable/key missing) |
model1_sample, etc. | Separate sample markers for each model; do not use in_analysis as a substitute |
dup_group | Duplicate group ID; keeper = 0, copy = 1 |
exclusion_reason | Reason for exclusion, useful for writing sample flow |
Use scripts/csv_to_spss_syntax.py to generate .sps, avoiding column type mismatches when manually writing GET DATA. If the CSV has binary text values like "Yes/No, Present/Absent, Support/Oppose", the script will automatically append RECODE ... ('Yes'=1) ('No'=0) INTO *_num; logistic regression can then use the numeric column; add --no-recode if not needed.
Gate: CSV row count, JSONL row count, SAV row count, and the unique sample count from sample_flow.md must all match; each model's N must be recalculable from the filter conditions.
Stage 4: Statistical Analysis & Expansion
Write the analysis plan first, then look at results. Default is a combination of "primary analysis + description + bivariate + multivariate + diagnostics + heterogeneity + robustness," not just running one primary model.
First, run the automated screening script to produce descriptive statistics, between-group tests, correlations, and univariate models:
python scripts/auto_stats.py --data data/analysis.csv --config assets/chart_config.example.json --out output/analysis_pack
It outputs descriptives.csv, group_tests.csv, binary_rates.csv, correlations.csv, univariate_models.csv, univariate_skipped.csv, and analysis_pack.md to identify variables worth further modeling; final models are still estimated with predetermined specifications in SPSS/R/Stata; do not use the automated screening p-values to select variables.
The script automatically identifies outcome variable type:
- 0/1 → binary outcome (Logistic/OR).
- Continuous numeric → continuous outcome (OLS/mean+CI); for multi-category, use Kruskal–Wallis for between-group tests.
- Multi-category → categorical outcome; use chi-square/Cramér's V for between-group tests; multi-category independent variables enter univariate OLS/Logit as dummies, reported as
variable_level. - Multi-category outcomes themselves are not run as binary Logit; the main model should use multinomial Logit or merge categories first.
- Common text labels like "Yes/No, Present/Absent, Support/Oppose" are automatically mapped to 0/1; other binary text must specify the positive class using
outcome_mapin the config. - Completely separated or unstable univariate models are written to
univariate_skipped.csv, not reported with misleadingly huge ORs. - If the outcome column is entirely missing, zero rows after filtering, or
filter_queryis misspelled, it exits with an error.
Minimum content:
- Descriptive statistics: N, proportion/mean, outcome rate, 95% Wilson CI for each group.
- Main model: binary Logistic / OLS / multinomial Logit, specifying dependent variable, reference group, key independent variable, control variables.
- Mechanism/moderating model: interaction term, simple slope, or grouped predicted values.
- Robustness: alternative outcome, alternative sample, add/remove control variables, before/after deduplication.
- Diagnostics: multicollinearity, complete separation, underpowered cells, calibration/discrimination.
- Heterogeneity: by category/region/year/population.
- Exploratory analyses that can be done: PCA, correspondence analysis, clustering, predictive ROC/calibration; label as "exploratory."
Tables must include at least: coefficient/OR, 95%CI, p, N, fit index, reference group description. Each model has its own row for N.
Write all conducted and skipped analyses into analysis_manifest.md (template at references/08-analysis-expansion.md); do not report only significant results.
SPSS syntax templates and interpretation guidelines are in references/03-analysis-spss.md; expansion analysis checklist is in references/08-analysis-expansion.md.
Gate: Any OR/β must be traceable from analysis.sps → output files → table in three steps; analysis_manifest.md explains why each analysis was done or skipped and the sample size; the body text does not make causal claims from "the text mentioned a certain situation" to "the situation actually occurred."
Stage 5: Publication-Grade Charts (Default: Entire Portfolio, Not Just 8 Charts)
First, run scripts/auto_chart_pack.py to generate the full chart portfolio, then select main body figures, appendix figures, and any additional model-based charts:
python scripts/auto_chart_pack.py \
--data data/analysis.csv \
--config assets/chart_config.example.json \
--out output/chart_pack --tiff
For continuous/multi-category outcomes, no script changes are needed: the outcome variable automatically switches to "outcome distribution + grouped mean/composition" charts; text outcomes can use outcome_map for encoding. --coef supports metric (OR/HR/RR/beta) and n columns; forest plots write N based on the model sample size. --pred supports 0/1 or "Yes/No" labels; constant predictions or single-category outcomes skip ROC/PR/calibration with an explanation. term_labels can map model term names like lv3 to readable labels. Two-proportion dumbbell plots require specifying two 0/1 columns in the config under dumbbell. For survival data with follow-up time and event columns, use --survival-time and --survival-event (optionally --survival-group) to generate Kaplan–Meier curves.
Default coverage: 7 groups, typically around 40–55 charts: ~44 for binary (with model input ~51), ~42 for continuous, ~41 for multi-category. The actual count varies with column types, missingness, and skipped items; the authoritative count is in chart_manifest.csv.
- A Data Quality: Missingness rate, sample category/year/region, binary variable incidence.
- B Univariate: Outcome distribution (binary/continuous/multi-category handled separately; continuous outcome adds Q-Q), treatment composition (bar + donut), numeric distributions, category composition.
- C Bivariate: Category x outcome, stage x outcome, category x stage heatmap/stacked, region/court-level differences, correlation heatmap, scatter fit, bubble plot; continuous outcome uses mean+95%CI; multi-category outcome uses within-group composition stacked charts.
- D Multivariate: Violin/boxplot, scatter matrix, dumbbell plot, PCA scree/biplot/loading heatmap, Ward clustering dendrogram, classic MDS.
- E Time Series: Outcome rate/mean trend, 3-sigma control chart, sample size, stage composition stacked bar/area; multi-category outcome adds outcome category composition trend.
- F Model Diagnostics: Forest plot, ROC, calibration, predicted distribution, confusion matrix, PR curve, decile lift (requires
--coef/--pred); Kaplan-Meier (requires survival columns). - G Appendix: Correspondence analysis, Pareto.
Charting strategy:
- At least 2 charts per primary finding: main result chart + robustness/diagnostic/heterogeneity chart.
- Body text: 7–10 charts; appendix: 20–50 charts; no upper limit.
- Each chart must first answer "What question?" If there is no question, do not draw it.
- Use
references/07-chart-atlas.mdfor chart selection, covering common SPSS advanced graphics: clustered/stacked/100% stacked bar, line/area, histogram, boxplot/violin, error bar, scatter/bubble, scatter matrix, Pareto, ROC, calibration, Kaplan-Meier (with survival data), scree plot, factor loading, cluster dendrogram, correspondence analysis, MDS, control chart, etc. - Each chart is recorded in
chart_manifest.csv: chart number, question, type, variables, N, caption, file path. - Color semantics are consistent throughout the manuscript: the same stage/category uses the same color in every chart.
- Chart captions include N, data source, model specification, reference group; exploratory charts (PCA/CA/clustering) are labeled "exploratory" in the caption.
Use scripts/pubfig.py for unified style:
from pubfig import set_style, save_fig, wilson_ci, forest_plot
set_style() # 15 cm column width, SimSun/Times New Roman, 600 dpi
Specification points:
- Physical width: 15 cm (within A4 body 15.4 cm); font size: 7.2–8.5 pt.
- Chinese font: SimSun (Song); Western font: Times New Roman; embed fonts in PDF (Type 42).
- No titles within charts; chart titles and captions are placed in Word.
- Output simultaneously: PNG 600 dpi, vector PDF, LZW TIFF (
--tiff). - Error bars: must state 95% CI; forest plots: solid dots = significant, hollow dots = not significant.
- Skewed continuous variables: apply log10 transformation or P99 truncation; indicate in axis label and caption.
Gate: Every chart in chart_manifest.csv has a question, N, caption, and file; main body charts and appendix charts do not overlap; embedded chart width in PDF ≥ 3000 px (approximately 600 dpi at 15 cm width); captions, charts, and body text citations are all present.
See references/04-figures-publication.md and references/07-chart-atlas.md for details.
Stage 6: Manuscript Assembly
Word layout requirements:
- Set charts, chart titles, and captions to "Keep with next" to prevent chart titles from splitting across pages.
- Table rows must not break across pages; repeat header rows on each new page.
- Table notes go below the table/figure; provide N first, then significance rules.
- When a figure/table is first cited in the body text, the numbering must match the file names (Figure 1, Table 1, Appendix Figure 1).
- N and percentages in the abstract, body text, tables, figures, and appendix must all come from the same source; the N from
chart_manifest.csvis written back into chart captions.
When exporting Word to PDF, Word compresses 600 dpi images to about 200 dpi. Use scripts/export_pdf_hires.py to export and then backfill the original images:
python scripts/export_pdf_hires.py --docx "manuscript.docx" --out-pdf "manuscript.pdf" --figure-map "figure_map.json"
Gate: PDF page count, chart count, and chart title count must all match; every embedded image width ≥ 3000 px; tables have no broken rows or split headers.
See references/05-manuscript-assembly.md for details.
Stage 7: Final Check
Run:
python scripts/qa_outputs.py --docx "manuscript.docx" --pdf "manuscript.pdf" --figures "figures" --data "data/analysis.csv" --audit "output/audit/corpus_audit.csv" --manifest "output/chart_pack/chart_manifest.csv" --strict
Then manually go through references/06-qa-checklist.md. Focus is not on formatting, but on:
- Every number in the sample flow must reconcile.
- Duplicate copies must be removed from the analysis sample, or their impact must be explicitly reported if not removed.
- Every N in the body text must match the corresponding table/figure/model.
- Charts and text must describe the same direction; no "chart goes up, text says down."
- Chart title numbers in the PDF must be identifiable (e.g., "Figure 1:", "Figure 1.", "Table 1:"); low-resolution check only looks at images that are large enough; empty CSVs cause an immediate error.
- The source of theoretically coded variables must be explained in the methods section; do not write as if they were directly measured from raw materials.
- Conclusions use "association/correspondence," not causal language.
Scripts Index
The following commands assume the current directory is the skill root; if running from the project directory, replace scripts/ with the absolute path to the skill.
| Script | Purpose |
|---|---|
scripts/audit_corpus.py | Scan any materials directory: file status, text/pages, MD5, duplicates, format distribution, year hints |
scripts/dedupe_corpus.py | Generate a deduplication plan based on audit results; --apply moves copies to _duplicates/ |
scripts/make_coding_sheet.py | Extract full text and keyword evidence sentences from PDF/Word/PPT/HTML/text; --profile legal also extracts court/case number fields |
scripts/text_extract.py | General text extraction and file checking (PDF/DOCX/ODT/PPTX/XLSX/HTML/text/images/audio/video) |
scripts/analysis_utils.py | Internal shared functions: sample filtering, outcome type identification, binary label encoding; imported by automatic statistics/chart scripts |
scripts/csv_to_spss_syntax.py | Generate SPSS GET DATA syntax from CSV headers; clean illegal/duplicate variable names and write variable labels |
scripts/auto_stats.py | Automatic descriptive statistics, between-group tests, correlations, univariate models (dummy coding for multi-category independent variables), complete separation check |
scripts/auto_chart_pack.py | Automatically generate ~40–55 data quality/distribution/bivariate/clustering/time/model diagnostic/survival charts and manifest based on column roles |
scripts/pubfig.py | Unified publication chart style, Wilson CI, forest plot, heatmap, violin, ROC/PR, PCA, MDS, clustering, KM, control chart, and 30+ chart types |
scripts/export_pdf_hires.py | Word→PDF with 600 dpi image backfill |
scripts/qa_outputs.py | Check PDF image resolution, docx chart/table count, empty data, duplicate keys, manifest N/caption/path |
Reference Documents
| File | When to Read |
|---|---|
references/01-corpus-audit.md | When starting the audit, handling duplicates/corrupt scans/multiple formats |
references/02-coding-protocol.md | When designing variables, writing codebook, performing reliability checks |
references/03-analysis-spss.md | When specifying models, writing SPSS syntax, interpreting ORs |
references/04-figures-publication.md | When creating charts, selecting colors, exporting submission files |
references/05-manuscript-assembly.md | When formatting Word, exporting PDF, handling chart captions |
references/06-qa-checklist.md | Before submission/delivery, checking each item |
references/07-chart-atlas.md | When selecting chart types, matching SPSS advanced graphics, determining chart count |
references/08-analysis-expansion.md | When deciding which additional analyses to run beyond the main model, documenting why each was done or skipped |
Templates
assets/scope-template.md: Research scope, inclusion/exclusion criteria, sample flow skeleton.assets/codebook-template.md: Variable coding manual skeleton.assets/figure-spec-template.md: Chart caption and specifications table.assets/figure_map.example.json: Example mapping for Word→PDF image backfill.assets/delivery-checklist.md: Delivery checklist.assets/coding-keywords.json: Replaceable keyword dictionary (generic placeholder example).assets/chart_config.example.json: Column roles, labels, analysis sample filter (filter_query) example.assets/analysis-manifest-template.md: Main/expansion/skipped analyses checklist.
Most Common Pitfalls
- Treating the number of files in a folder as the sample size. Must first subtract metadata, corrupt files, duplicates, unsupported formats, and out-of-scope items.
- Deduplicating only by filename. Same content often appears in multiple versions with/without identifiers, "- copy," "(2)".
- Mixing descriptive statistics N with model N. Models have missing values and selection criteria; N is always smaller.
- Treating "the material mentioned a certain situation" as "the situation actually occurred." This is a text association, not a causal effect.
- Word-exported PDF looks normal, but images are only 200 dpi. Must backfill and verify embedded image width.
- Drawing only one main result chart. Distribution, heterogeneity, robustness, and diagnostic charts were not drawn; user has to repeatedly ask.
- Treating content from citations/appendices as primary material. Document type, report type, web navigation must be judged by title/body structure, not by full-text keywords.
- Result is "Yes/No" text but treated as a continuous variable. Use
outcome_mapto define the positive class, or encode as 0/1 in the CSV; script will not guess positive/negative direction. - Auditing only one format, silently missing other files. Every file must have a status;
total_filesmust reconcile with the file manager. - Stuffing tabular data into text coding. CSV/Excel go directly through
auto_stats/SPSS workflow; text coding is for unstructured materials like documents, interviews, and web pages.