generate-questions

What does generate-questions do?

This skill produces pool.json: a list of candidate research questions, each with an anonymous id, merged so that near-duplicates appear once, and rewritten so that every question has the same plain style and names no direction for the effect it asks about. The next skills (screen-pilot, critique-forecast, check-novelty) take this file as input.

What you need first

• A field profile: a copy of field-profile/TEMPLATE.md from this package, filled in for your field and saved in the project (by default FIELD-PROFILE.md at the project root). The researcher should agree its content, in particular the field, the subjects the pilots will run on, the pilot limits and who would want to know the answers. The tested profile is field-profile/example-small-language-models.md.
• A work folder for this round, here called RUN/.
• Sub-agents with their own contexts. Where you can choose the model, use a different model for each generator. You may also run a generator through another provider's tool or API and save its output in the same format.

Steps

1. Choose the generators. Use at least three generating models, each in both modes, so each model runs two generators: plain and distant. In the test, three Claude models (Opus 5.5, Fable 5.2, Sonnet 5.5) each wrote 17 questions per mode.
2. Assign distant fields. Choose a list of distant fields with well-established findings (tested: cognitive psychology, psycholinguistics, behavioural economics, ecology) and draw one per distant generator without replacement: python3 scripts/assign_fields.py SEED "field one;field two;..." NAME1 NAME2 NAME3. Record the seed and the output.
3. Write the briefs. python3 scripts/make_briefs.py FIELD-PROFILE.md RUN N NAME1:plain NAME1:distant:"FIELD" ... writes one brief per generator in RUN/briefs/ (a short header, the mode's prompt from prompts/plain.md or prompts/distant.md, then prompts/constraints.md, all filled from the profile) and creates the output folders.
4. Run the generators. Start one fresh sub-agent per brief, with the model the brief is for, and tell it to read only its brief and follow it. Show it no example questions and no other generator's output. It writes candidates.json (and, in distant mode, transfers.md, its working list, which is kept for the record and not passed on).
5. Check the format. python3 scripts/check_candidates.py N RUN/generation/*/*/candidates.json. Ask a generator once for a fix if its file fails, and record that.
6. Pool and find near-duplicates.
   • python3 scripts/merge.py RUN/pool SEED plain:NAME1=RUN/generation/plain/NAME1/candidates.json distant:NAME1=... ... keeps only the question text and shuffles.
   • python3 scripts/candidate_pairs.py RUN/pool lists pairs of similar questions. The tested version embeds the questions with nomic-embed-text through a local Ollama server and keeps pairs with cosine similarity of at least 0.83, at most 5 nearest neighbours per question. Without Ollama, add --lexical (untested: word-count similarity, each question's 5 nearest neighbours).
7. Judge duplicates. Fill prompts/dedup_judge.md from the profile (python3 scripts/fill_prompt.py FIELD-PROFILE.md prompts/dedup_judge.md > RUN/pool/dedup_judge.md). For more than about 150 pairs, split them (python3 scripts/split_pairs.py RUN/pool 150) and give each batch to a fresh judge (tested: Sonnet 5.5, four batches of 152 pairs). Each judge reads only the filled prompt and its batch and writes its decisions; join them with python3 scripts/join_decisions.py RUN/pool decisions_1.json .... Then python3 scripts/apply_merges.py RUN/pool merges clusters and writes to_rewrite.json.
8. Rewrite. Fill prompts/rewrite.md the same way and give it, with RUN/pool/to_rewrite.json, to one fresh agent (tested: Sonnet 5.5), which writes RUN/pool/rewritten.json and reads no other file.
9. Anonymise. python3 scripts/anonymise.py RUN/pool SEED checks that every id was rewritten, lists questions over 50 words, reshuffles, and writes RUN/pool/pool.json (ids Q01, Q02, ... and text) and RUN/pool/sources.json (which generator and mode each came from). Show critics, forecasters, pilot builders and novelty checkers only pool.json.

Seeds: any fixed integers, chosen and recorded before the step runs.

What the test showed

In the test (field: the observable behaviour of small open language models), 12 generators wrote 204 questions in about three minutes. Merging left 168 (18% were near-duplicates; ideas such as stated confidence separating correct from incorrect answers came up in several generators). The share of questions whose pilot found an effect and whose literature check found no earlier report in the same direction was:

Mode	New and confirmed findings
Plain prompt	14 of 50 (28%)
Transfer from a distant field	14 of 51 (27%)
Problematization (not included)	6 of 42 (14%)
Gap mapping from recent abstracts (not included)	3 of 36 (8%)

With 36 to 51 questions per mode these differences are descriptive (test of equal rates, p = 0.054). The two modes not included are documented in NOT-RECOMMENDED.md at the top of this package.

Files

• prompts/plain.md, prompts/distant.md: the generation prompts. N is replaced by the number of questions; the other placeholders come from the field profile.
• prompts/constraints.md: constraints every generator receives.
• prompts/dedup_judge.md, prompts/rewrite.md: the duplicate judge and the rewriter.
• scripts/: fill_prompt.py, make_briefs.py, assign_fields.py, check_candidates.py, merge.py, candidate_pairs.py, split_pairs.py, join_decisions.py, apply_merges.py, anonymise.py. Standard library only.