johnsmithCA-sta/health-report-trend-analysis

Health Indicator Trend Analysis System

Purpose

Normalize multi-year, multi-format (electronic PDF / scanned document / photo) health checkup reports into a standardized indicator time series, producing a trend analysis report and an interactive health management dashboard. Fully local processing, automatic anonymization, and dynamic expansion (simply re-run the pipeline to add new reports).

Trigger Phrases

• Multi-year health checkup reports / annual health checkups
• Digitize checkup data / parse checkup reports
• Health indicator trends / indicator trend analysis
• Year-over-year indicator comparison / annual comparison / year-over-year changes
• Lab test analysis / lab test interpretation / test report analysis
• Scanned checkup reports / OCR scanned documents
• Checkup report photos / photo version checkup reports / smartphone photos of reports
• Health management dashboard / health data dashboard / visualization dashboard
• Blood glucose trends / blood glucose changes / high uric acid / lipid trends / blood pressure trends
• Abnormal indicators / abnormal indicator interpretation
• Add new checkup report / integrate new report / add a new year's checkup
• Anonymize checkup data / anonymized dataset / shareable dataset

Environment Setup

# Dependencies (skip if installed): pymupdf is mandatory; Pillow (photo compositing), pyobjc-framework-Vision/Quartz (OCR, macOS only) are optional
export REPORT_DIR=/path/to/checkup-reports      # Report directory (containing PDFs or year subdirectories with JPGs)
export WORK_DIR=/path/to/work-data-directory    # Working data directory (auto-creates data/ and output/)
export PY=/path/to/python                       # Use a Python interpreter with dependencies installed

Step 0 (mandatory at the start of each new session):

$PY scripts/check_deps.py            # One-click dependency check: shows what's missing, which scripts are affected, and how to install
$PY scripts/check_deps.py --json     # Machine-readable output for automation

Exit code 0 = mandatory dependencies ready; 1 = missing mandatory dependencies (main pipeline blocked, install as prompted); missing optional dependencies only affect the corresponding input modality but do not block the electronic PDF main path.

Execution Flow

All scripts support --help; parameters take precedence over environment variables; running without parameters uses existing environment variable behavior.

1. Identify Input Modality: Traverse REPORT_DIR:
   • *.pdf with text layer in root directory → electronic version (processed by parse_reports.py)
   • *.pdf without text layer in root directory → scanned document (vision_ocr.py + extract.py)
   • Year sub-directory YYYY_health_report/*.JPG → photo version (make_year_pdfs.py compositing → OCR → extract.py)
2. Parse Electronic PDFs (coordinate-based extraction; for column boundaries, see references/parsing_and_reference_ranges.md):

   $PY scripts/parse_reports.py                        # → data/reports_raw.json
   $PY scripts/parse_reports.py --year 2027 --dry-run  # Single-year dry run, no writes
   

3. Photo PDF Compositing + OCR + Extraction (if JPG directories or scanned documents exist):

   $PY scripts/make_year_pdfs.py --year 2015-2020      # Multiple JPGs per year → single PDF (horizontal photos auto-rotated)
   $PY scripts/vision_ocr.py <pdf> data/ocr_2022.json  # Scanned/photo OCR → JSON with coordinates
   $PY scripts/extract.py --year 2022                  # OCR results → indicators (strategy auto-detected)
   $PY scripts/extract.py --years 2015-2020            # Multi-year dict-type OCR (abbreviations + reference ranges inferred)
   

   extract.py is parameterized by year; no code changes needed for new years: place data/ocr_<year>.json and run --year <year>. The default strategy is auto (top-level JSON is dict → abbr, is list → row); use --strategy to override.
4. Normalize + Build Dataset (name/unit mapping, dirty data filtering, automatic OCR injection, generate anonymized version):

   $PY scripts/build_dataset.py
   

   Outputs data/dataset_std.json (full version, locally controlled) + data/anonymized/anonymized_dataset_anon_shareable.json (anonymized version, separate subdirectory + read-only 444 + shareable)
5. Trend Analysis (flag abnormal indicators based on current year's reference ranges, significant changes ≥20% or ≥30% of reference width, improving/worsening trends, focus on recent three years):

   $PY scripts/trend_analysis.py                        # Default: last three years
   $PY scripts/trend_analysis.py --focus-years 2020,2023,2026
   

6. Generate Deliverables:

   $PY scripts/report_generator.py    # output/health_indicator_trend_analysis_report.md (default: **enables** body text anonymization)
   $PY scripts/build_dashboard.py     # output/health_dashboard.html (single-file offline)
   

New Report Integration Guide

When adding a new year's health checkup report, follow the appropriate branch based on modality (see references/new_report_integration_guide.md for details):

1. Place the report: *.pdf in REPORT_DIR root, or create a year directory YYYY_health_report/ with JPGs
2. Identify modality: Check PDF text layer (get_text() length >0 = electronic version)
   • Electronic version → directly run parse_reports.py; if column boundaries are unexpected, first print a one-page coordinate sample
   • Scanned/photo → make_year_pdfs.py → vision_ocr.py → extract.py --year <year> (do not create a new year-specific script)
3. Update dictionary (if unrecognized indicators): Add one mapping line to NAME_MAP in indicator_dict.py (see logs in data/unrecognized)

   $PY scripts/indicator_dict.py --search uric_acid    # Check if already recorded (including aliases)
   $PY scripts/indicator_dict.py --validate            # Self-check dictionary integrity (dangling aliases/duplicate keys)
   $PY scripts/indicator_dict.py --count               # Current total standard indicators
   

4. Re-run pipeline: build_dataset → trend_analysis → report_generator → build_dashboard
5. Verify: Confirm the new year appears in trend_analysis.json["years"] and on the dashboard timeline; spot-check 3 key indicator values against the original report; generate and verify the anonymized version

Key Rules

• Privacy Hard Constraint (zero tolerance): Fully local; no external network calls within the skill; must generate and verify the anonymized version upon task completion (no names, ID numbers, phone numbers, addresses, or hospital names); only share data/anonymized/anonymized_dataset_anon_shareable.json (strictly separated from the full dataset_std.json in a different directory, read-only 444, preventing accidental sharing of the full version)
• Anonymization verification before writing to disk: _write_anon_verified() in build_dataset.py is the sole entry point for writing the anonymized version—verification occurs in memory first; only after passing is it written to disk. The anonymized version is set to read-only 444 upon writing. If verification occurred after writing, a file with PII would already be on disk and unchangeable, with [FAIL] being merely a post-hoc warning. Do not bypass this entry point by calling _write_anon() directly; --skip-verify is for troubleshooting only, and the resulting artifacts must not be shared.
• Report Body Anonymization: mask_identity_text in report_generator.py is enabled by default (masks institution names, doctor names, phone numbers, ID numbers); use --no-mask to disable, for local cross-checking only—do not share the generated artifact.
• Adding new annual reports: Place in REPORT_DIR → re-run steps 2→6; unrecognized indicators are logged in data/unrecognized; add one line to NAME_MAP in indicator_dict.py to resolve.
• OCR Data Usage: Indicators extracted via OCR are used only for chart display (marked with a gray Ⓞ label) and do not participate in significant change determination to avoid polluting statistics.
• Trend Focus: Overall trends and first-to-last comparisons are based only on focus years (default: last three years, overridable with --focus-years); historical data serves as background context only.
• Reference Range Consistency: Abnormal determinations use the reference range from the current year's report (automatically adapts to reagent changes, e.g., direct bilirubin from 2025 onward: 0.0–4.0 → 1.7–6.8).

Deliverables

• output/health_indicator_trend_analysis_report.md — Indicator summary table, abnormal highlights, authoritative interpretations, tiered recommendations (lifestyle/diet/exercise/medical consultation indicators), methodology
• output/health_dashboard.html — Health overview, improving/worsening group filtering, line charts with reference range bands, time span switching (single file, no external dependencies)
• data/anonymized/anonymized_dataset_anon_shareable.json — Shareable anonymized dataset

Script Overview

Script	Purpose	Key Parameters
check_deps.py	Dependency self-check (Step 0)	--json --quiet
parse_reports.py	Electronic PDF parsing	--report-dir --data-dir --year --out --dry-run
make_year_pdfs.py	Yearly JPG → PDF	--report-dir --year --out-dir --dry-run
vision_ocr.py	Scanned/photo OCR	pdf out (positional) --dpi --quiet
extract.py	OCR results → indicators (parameterized)	--year --strategy --ocr-json --out --data-dir --quiet
build_dataset.py	Normalization + anonymization	--work-dir --data-dir --out --skip-verify --dry-run
trend_analysis.py	Trend determination	--work-dir --focus-years --out --dry-run
report_generator.py	Markdown report	--work-dir --out --no-mask --dry-run
build_dashboard.py	HTML dashboard	--work-dir --data-dir --out --quiet
indicator_dict.py	Indicator dictionary (imported)	--list --search --count --validate --category (effective only when run directly)

Exit codes are tri-state: 0 success / 1 runtime failure / 2 parameter or precondition failure.

Reference Documents

Document	Content
references/new_report_integration_guide.md	Complete steps and troubleshooting for new modality/year report integration
references/parsing_and_reference_ranges.md	Column boundary coordinates and reference range definitions for electronic PDFs
references/OCR_methodology.md	OCR strategy for scanned/photo documents, abbreviation mapping table maintenance
evals/	8 regression tests (including 3 privacy defense tests); run with eval_loop.py --run

Usage Example (Full Session)

User: "Help me add the 2027 health checkup report and update the trend analysis."

1. Step 0: python3 scripts/check_deps.py → mandatory dependencies ready
2. Check REPORT_DIR: find new file health_report_270510004567.pdf (electronic version, text layer complete)
3. Small sample column boundary check: print 1-page coordinates, consistent with references/parsing_and_reference_ranges.md → no adjustment needed
4. Run pipeline:

   export REPORT_DIR=health_reports WORK_DIR=/path/to/work
   python3 scripts/parse_reports.py
   python3 scripts/build_dataset.py
   python3 scripts/trend_analysis.py
   python3 scripts/report_generator.py
   python3 scripts/build_dashboard.py
   

5. Parse logs show: new year has 98 indicators, 1 unrecognized name → indicator_dict.py --search confirms not recorded; add 1 line to NAME_MAP and re-run build_dataset
6. Verify: trend_analysis.json["years"] contains 2027; dashboard timeline shows 2027; spot-check blood glucose/lipids/BMI values against the report; grep the anonymized version shows no identity information
7. Deliver: Provide the report and dashboard paths to the user; explain changes and recommendations for each abnormal indicator in 2027

Cost Discipline (Enforced at Execution)

1. Small sample first: Before parsing a new format, print a 1-page coordinate sample to confirm column boundaries, then process the full file (avoid full file dumps)
2. Output compression: Debug commands unify | head -N / python -c for precise extraction; do not cat large JSON files in full
3. Sub-agent isolation: Large file reading/format detection is handed off to the Explore sub-agent; the main thread only receives conclusions
4. Verification: Take only one final screenshot; use DOM/JS checks for intermediate iterations

Changelog

Version	Date	Change
1.1.0	2026-08-29	Engineering consolidation overhaul: ① All 10 scripts now support --help + argparse + tri-state exit codes (previously 0/10, worst in class); ② Added check_deps.py dependency self-check as Step 0; ③ Hardcoded year-specific extract_2015_2020.py / extract_2022.py merged into parameterized extract.py --year (new year integration no longer requires code changes, output format field-by-field identical to old scripts); ④ Added ## Trigger Phrases section (12 items) + evals/ 8 regression tests (including 3 privacy defense test cases); ⑤ Privacy fix: Anonymized version changed to "verify in memory first, write to disk only if passed" (previously verified after writing—when PII was hit, the file with identity data was already on disk and read-only, with [FAIL] being a mere post-hoc warning); ⑥ Fixed dead code in trend_analysis.py that lost measurement items (measurement_items was reset after appending; changed to deduplicate by key)
1.0.3	2026-08-23	P2 M-1 fix (skill safety): Strict isolation of anonymized version—anonymized_dataset.json → data/anonymized/anonymized_dataset_anon_shareable.json (separate subdirectory + read-only 444 + _anon_shareable naming marker), preventing mix-up and accidental sharing of full version with anonymized version; build_dataset.py added _write_anon() (tmp+replace atomic write + chmod 444); report_generator privacy statement paths synchronized
1.0.2	2026-08-17	P1 fix: Added verify_anonymized auto-verification at end of build_dataset.py (regex scan for ID numbers/phone numbers/email/hospital names; exits with error on hit)
1.0.1	2026-08-17	P0 fix: report_generator.py added mask_identity_text anonymization function (masks phone numbers/ID numbers/doctor names + generalizes institution name); anonymized summary/conclusion in text display layer; privacy statement aligned with actual output
1.0.0	2026-08-16	Initial release: Health Indicator Trend Analysis System