mining-data-preparation
Solve the problem of "files can be opened, but data cannot be directly compared," allowing multi-source records to enter subsequent analysis while preserving on-site changes.
Core Capabilities
- Multi-source alignment: Associate instrument outputs, measurement point ledgers, and excavation logs by time, space, and object.
- Evidence-based cleaning: Identify duplicates, missing measurements, drift, and unit issues, avoiding the elimination of genuine engineering anomalies as noise.
- Auditable processing: Preserve original values, transformation rationale, and record mappings, supporting review and reprocessing.
Getting Started
First, understand whether the data will be used for plotting, statistical comparison, anomaly detection, or modeling. Then, check the file structure, field meanings, and coverage. Processing granularity is determined by the analysis goal; follow existing data standards when available rather than forcing conversion to a unified template.
Workflow
- Identify record granularity, field meanings, and downstream analysis purpose.
- Use local profiling tools to inspect fields, missing values, duplicates, and time coverage; then interpret issues in the context of operating conditions.
- Transform, associate, or clean on a copy based on clear rules, preserving original values and record mappings.
- Review the processing impact, deliver usable data, and identify remaining issues that still limit analysis.
On-Demand Resource Usage
- Before using local tools, read Parameters and Scope, then call scripts/profile_csv.py. The tool only processes the described formats and capabilities; it does not imply the complete task is done.
- For an initial understanding of input formats, view the synthetic example examples/monitoring.csv; demo data should not be treated as actual measurements or standard parameters.
- For method selection, boundary judgment, or complex scenarios: read references/methods.md.
- When needing to quickly organize deliverables, copy and trim assets/data-dictionary.csv; this is an optional starting point, not a fixed output format.
Deliverables
When actual files are present, deliver an analyzable data copy along with necessary field descriptions, issue markers, and concise processing records. If only a review is needed, focus on the main issues affecting analysis and suggested corrections. Output format follows the user's workflow; do not generate multiple data sets redundantly for display purposes.
Cleaning completion does not mean the engineering status is normal; anomalies in the original records should remain for subsequent analysis and review.