benno-mueller/survey-analysis

Survey Analysis

Turn a raw survey export into findings a professional survey researcher would sign: every number has its base, uncertainty is stated honestly, differences are only claimed when they hold up, and no respondent can be identified.

The skill is tool- and topic-agnostic. Everything specific to one survey is a parameter (Phase 0), not a hard-coded assumption.

Ground rules (apply to every survey, override any other wish for "insight")

1. Statistical interpretation only. Describe what the data show: size, precision, robustness, differences, consistency. Never offer explanations, causes, motives or hypotheses about why a result is as it is - not even hedged ("possibly because ...", "this could indicate ..."). Respondents' own stated reasons (open answers) may be reported as what respondents said.
2. Equal weights by default. Every response counts once. Weights are used only if the survey design requires them (e.g. disproportionate sampling, a weight variable delivered with the data) or the parameters specify them. Never weight on your own initiative; report imbalances as a limitation instead.
3. Cross-tabulations and subgroup reports only where permitted. Whether results may be broken down at all, by which variables, and under which limits is a decision of the survey owner (crosstabs, subgroup_reports, anonymity parameters). Without permission: totals only.
4. All arithmetic in code (Python with pandas/scipy, or R). Never estimate a figure in prose. Work on a copy; never modify the user's original data.

Bundled files

File	Use it in
references/methods.md	Phases 3-9: formulas, test selection, weighting, disclosure control, worked example
references/reporting.md	Phase 10: management summary, action fields, structure, wording, charts
references/data-sources.md	Phase 1: export formats (LimeSurvey, MS Forms, Google Forms, SurveyMonkey, SPSS/R)
scripts/survey_stats.py	All phases: tested helpers - CIs with finite population correction, worst-case nonresponse bounds, tests and effect sizes, p-value adjustment, banner tables with significance letters, minimum detectable difference, raking and design effect, group disclosure plan, cell suppression, Cronbach's alpha, quality flags
assets/survey-config.template.yaml	Phase 0: the parameter file

If only this SKILL.md is installed, the section Core rules at the end is the minimum you need; implement the formulas yourself in code.

---

Phase 0 - Parameters

Look for a filled survey-config.yaml (template in assets/). Otherwise collect the parameters below. Facts that can be read from the data or files, read yourself; decisions belong to the user. Ask open decisions in one round (max. 4 questions), each with your recommended default. If nobody is there to answer, use the defaults and list them under "Assumptions" in the report. Defaults are deliberately restrictive: no breakdowns, no weighting.

Parameter	Meaning	Source	Default
population.definition	Who the survey is about	ask	- (required)
population.size_N	Size of the target population	ask	null = unknown/very large
population.design	census, simple_random, stratified, quota, self_selected	ask	self_selected (most conservative)
population.known_margins	Known population counts per characteristic	ask / files	none
fieldwork.invited / completes / partials / ineligible	Outcome counts	data + ask	completes = records with a submit date
purpose	describe, compare_groups, test_hypotheses, track_waves, benchmark	ask	describe
hypotheses	Pre-specified statistical comparisons (confirmatory)	ask	none -> all comparisons exploratory
crosstabs.mode	none, specified (listed variables), banner (full table band)	ask	none
crosstabs.variables	Breakdown variables that may be used	ask	[]
crosstabs.forbidden_combinations	Variable pairs that must never be crossed	ask	all two-variable combinations of demographics
subgroup_reports.enabled	Separate reports per unit (site, department, ...)	ask	false
subgroup_reports.unit_variable / min_n / min_response_rate / compare_to_total	Limits for unit reports	ask	- / anonymity.min_group_n / none / true
anonymity.min_group_n	Minimum respondents for any reported group	ask / agreement	5
anonymity.min_cell_n	Suppress cells with 0 < count < value	ask / agreement	null (off)
weighting.mode	none, design (weight variable in data), raking	survey design / parameters	none
weighting.variable / margins / trim	Details for design / raking	parameters	- / - / [0.3, 3]
analysis.confidence_level / alpha / power		default	0.95 / 0.05 / 0.80
analysis.multiple_testing	holm, bh, none	default	holm confirmatory, bh banner/exploratory
analysis.dont_know / dont_know_codes	Treatment of "don't know / no answer"	default	excluded from base, reported separately
analysis.likert_top_box	Categories counted as agreement	default	2 on 4-6 pt scales, 3 on 7+ pt
analysis.item_polarity	Which pole of each item is favourable	codebook + ask	from labels
analysis.quality_rules	Speeding ratio, straightlining min items	default	0.4 x median; 5 items
open_text.quotes	none, paraphrase, anonymized_verbatim	ask	paraphrase
recommendations.mode	none, action_fields, action_fields_with_measures	ask	action_fields
recommendations.measures	Measures supplied by the user to map onto findings	ask	[]
benchmarks	Earlier waves / external studies with identical wording	ask	none
output.audience / format / language		ask / conversation	lay / environment default / user's language

Phase 0 ends with a short parameter summary the user can correct.

Phase 1 - Ingest and codebook

Read references/data-sources.md for the export at hand. Then:

1. Load the data; drop direct identifiers immediately (IP, e-mail, name, token, referrer URL, exact timestamps if not needed) and say that you did.
2. Build a codebook: variable, question text, type (single choice, multiple choice, Likert k-point, numeric, open text, matrix row, metadata), value labels, scale direction and polarity, reversed items, missing/"don't know" codes, filter condition (who was shown the question).
3. Take scale direction and reversed items from the labels, never from assumption. If labels are missing, ask.
4. Recode into an analysis file; keep the mapping in the log.

Phase 2 - Data cleaning

Flag, document, and only then decide. Never delete silently.

• Completes vs. partials; define the analysis sample (default: completes; partials only for the items they answered, if the user agrees). Breakoff by page.
• Duplicates (same token, or identical answer vectors with near-identical timestamps).
• Speeders: total duration < ratio x median. Straightliners in matrices of

  = min_items items. Exclude only after showing the user count and pattern; never exclude on one flag alone when n is small.

• Item nonresponse per item; flag > 10 %.
• Out-of-range values, filter violations, impossible combinations.

Output: a data-cleaning log (check, count, decision, reason).

Phase 3 - Inference frame and weighting

Read references/methods.md, sections 1 and 7. The design decides what uncertainty statements are legitimate:

Situation	What to report
Census or near-census of a known, small population (n/N > 5 %)	CIs with finite population correction (FPC) plus worst-case nonresponse bounds for key figures. Phrase results as "among respondents" unless nonresponse checks support generalising.
Probability sample	Standard CIs (FPC if n/N > 5 %); design weights if the design has unequal selection probabilities.
Self-selected / open link / convenience	No population inference. Shares descriptively; CIs, if shown at all, only as "statistical precision", with an explicit caveat.

Weighting follows weighting.mode only (ground rule 2). If weights are used: report weighted and unweighted key figures side by side, the Kish design effect and effective n (kish_deff, effective_n), and use effective n for CIs and tests.

Phase 4 - Distributions of all answers

• Every item, every figure with its base (n) after filters and exclusions.
• Single choice: counts and shares; CIs (Wilson, with FPC if applicable) for key figures.
• Likert: full distribution first, then top-box / bottom-box with CI, median. Means +/- SD only as a supplement, and only for >= 5-point items or multi-item scales.
• Multiple choice: share of respondents per option; totals exceed 100 %.
• Numeric: median and IQR first, then mean/SD; check outliers.
• Rounding: whole percent when n < 200. Totals may not add to 100 % - say so once.
• Key figures vs. 50 % (or another reference value) only with mdd_one_proportion in mind: a share within the CI of 50 % is "about half".

Phase 5 - Cross-tabulations and significance

Governed by crosstabs.mode. With none, skip this phase entirely and say so in the method box.

1. Disclosure first. Use only crosstabs.variables; never cross forbidden_combinations. Run reportable_groups() for each breakdown: groups below anonymity.min_group_n are dropped, plus secondary suppression when the dropped remainder could be derived from the total. Apply anonymity.min_cell_n if set.
2. Power. For each breakdown compute the minimum detectable difference (mdd_two_proportions, 80 % power). Report it, so that "no significant difference" is read correctly ("differences below x points cannot be detected").
3. Tests. Choose by data type (references/methods.md, section 3): chi-square / Fisher for categories, Mann-Whitney / Kruskal-Wallis for ordinal items, Welch for metric scale scores. Always with effect size and its label; a p-value alone is never a finding.
4. Banner tables (crosstabs.mode: banner): banner_table() / format_banner() for every item x permitted variable - column percentages, base row, significance letters (a letter marks a significantly higher share than the column with that letter), letters based on p-values adjusted within the row. The band is exploratory: before a difference goes into the text, adjust all p-values of the band together with Benjamini-Hochberg (p_adjust over the returned tests). Banner tables go into the appendix; only robust differences (significant after band-wide adjustment and at least a small effect) go into the text.
5. Separate confirmatory results (pre-specified hypotheses) from exploratory ones.
6. A difference that is not significant is "no reliable difference", never a trend.

Phase 6 - Subgroup reports

Only if subgroup_reports.enabled. For each unit of unit_variable:

1. reportable_groups(sizes, min_n, total_published=True, invited, min_response_rate) decides which units get a report. The remainder rule matters: if one small unit is withheld but all others and the total are published, its result can be computed by subtraction - the function withholds further units until the withheld remainder is itself >= min_n. List withheld units and the reason (without their figures) for the survey owner, not in the published reports.
2. Each unit report uses the same structure as the main report, the unit's own base and CIs (FPC with the unit's invited count as N), and - if compare_to_total - the total for reference. Do not compare units with each other unless crosstabs permits that variable.
3. Inside a unit report no further breakdowns, unless explicitly permitted and all groups pass the thresholds.

Phase 7 - Scales and indices

Only for item sets designed as a scale (or validated instruments). Recode reversed items, compute Cronbach's alpha (and McDonald's omega if a factor package is available), item-rest correlations, alpha-if-deleted. Build a scale score only if alpha >= .70 (>= .60 for 3-item short scales, with a caveat).

Phase 8 - Open answers

See references/methods.md, section 6. Read every answer, build an inductive code frame, code with multiple codes allowed, report themes as "x of n respondents mentioned ...". Second coding pass on >= 20 %, report agreement. State that coding was done by an AI as a single coder. Quotes only as allowed by open_text.quotes, stripped of anything identifying. Report what respondents wrote; do not add your own explanation of it.

Phase 9 - Nonresponse and representativeness

• Response rates (RR1 / RR2, response_rates()), overall and per known group.
• Compare respondent composition with known_margins (goodness-of-fit test). Response rates differing by > 15 points between groups are reported as a limitation (and, if subgroup reports are permitted, key figures per group). No weighting unless weighting.mode says so.
• Item nonresponse patterns; early vs. late respondents if timestamps exist.
• Conclude explicitly to whom the results apply and with which limits.

Phase 10 - Interpretation, management summary and report

Read references/reporting.md.

• Interpretation = statistical reading only (ground rule 1): how large, how precise, how robust (CI, worst-case bounds), how consistent across permitted groups, how it compares with benchmarks of identical wording.
• Management summary (first page): 3-7 key findings with n and uncertainty, then action fields.
• Action fields (recommendations.mode): rank items or topics by statistical criteria only - share of unfavourable answers, robustness under worst-case bounds, consistency across permitted groups, share of "don't know". action_fields_with_measures: map measures the user supplied onto the findings they address; do not invent measures and do not predict their effect. Methodological recommendations (item revision, follow-up survey, larger base) are always allowed.
• Report body: method box, assumptions, results by theme, limitations, appendix (all frequency tables, test table, banner tables, codebook, cleaning log). Deliver in output.format using the environment's normal output mechanism.

Phase 11 - Verification (before delivery)

• Recompute 3-5 key figures independently (second script or manual counts).
• Every percentage has a base; bases match the codebook filters.
• No group or cell below the anonymity thresholds, including by subtraction across tables and subgroup reports.
• No breakdown that the parameters do not permit.
• No quote or detail that could identify a person.
• Every claimed difference: test, effect size, adjusted p-value.
• No explanatory hypotheses, no causal language; weights only as parameterised.
• Assumptions and exclusions are listed.

---

Core rules (self-contained minimum)

• FPC: SE x sqrt((N-n)/(N-1)). Apply when n/N > 5 % and N is known.
• Share CI: Wilson score interval; with FPC use n_eff = n / FPC^2.
• Margin of error (p = .5): 1.96 x sqrt(0.25/n) x FPC.
• Worst-case nonresponse bounds (census): [k/N, (k + N - n)/N].
• Minimum detectable difference (two groups, 80 % power, alpha .05): 2.80 x sqrt(0.25 x (1/n1 + 1/n2)).
• Categories x groups: chi-square; Fisher exact for 2x2 with expected < 5; permutation p if > 20 % of expected counts < 5. Cramer's V (.10/.30/.50 for df=1).
• Ordinal x 2 groups: Mann-Whitney U + Cliff's delta (.15/.33/.47); 3+ groups: Kruskal-Wallis + epsilon^2.
• Significance letters: pairwise two-proportion tests per row, adjusted within the row; letter X = significantly higher than column X.
• Multiple tests: Holm (confirmatory), Benjamini-Hochberg (exploratory, banner).
• Weights: none by default. If parameterised: raking to margins, trim, report Kish deff = n x sum(w^2) / sum(w)^2 and effective n = n / deff.
• Disclosure: no group with n < min_group_n; withheld groups together must reach min_group_n when the total is published; optional cell suppression.
• Reliability: alpha = k/(k-1) x (1 - sum item variances / variance of sum).

Anti-patterns (never do this)

• Explaining results ("this is probably due to ...", "a possible reason is ...").
• Weighting because the sample "looks unbalanced" without a parameter for it.
• Breakdowns or crosses the survey owner did not permit.
• Publishing all units but one, together with the total.
• Calling 52 % "a majority" when the CI includes 50 %.
• Reporting the mean of a 4-point scale as the headline figure.
• Percentages for groups of 7 people, or bars without n.
• Reporting significant hits from a banner band as findings without adjustment and effect size.
• "No difference" without stating the minimum detectable difference.
• Calling a survey "representative" because the response rate is high.
• "The employees think ..." when only respondents answered.
• Recommending measures the data cannot support, or predicting their effect.