htlin222/ask-oe-with-ego-skill

Ask OpenEvidence with ego-browser

Two layers, deliberately split:

1. Capture — needs ego (scripts/dump.mjs). Reuses the user's real, DataDome-trusted OpenEvidence session to ask/publish and snapshot the rendered SPA DOM into oe-<id>.dump.json. This is the only step that touches openevidence.com. A headless Playwright/Puppeteer would trip DataDome here (webdriver + fingerprint + datacenter IP + no clearance cookie) — ego works because it is the user's trusted browser.
2. Transform — fully standalone (scripts/oe-transform.mjs). Plain Node (≥18, global fetch), no ego, no browser. Reads a dump, resolves every reference through the Crossref API (PMIDs converted via NCBI first), and writes oe-<id>.html + .bib + .csl.json + .ama.txt. Only outbound calls are Crossref + NCBI (open APIs, no DataDome).

The HTML is thread-aware: one card per turn, nested TOC + scrollspy, citation popovers (AMA text), reference back-links with ×N counts, reading-progress bar, theme toggle, and toolbar ⬇ .bib / ⬇ CSL / ⬇ AMA downloads. A single answer is a 1-turn thread — same pipeline.

Preflight — is ego installed?

The capture step needs the ego-browser CLI. Check first:

command -v ego-browser >/dev/null 2>&1 && echo "EGO OK" || echo "EGO MISSING"

If EGO MISSING, guide the user through installing ego-lite following references/install.md — briefly: download the Mac build from https://lite.ego.app/download?auto=1, drag it into Applications, launch and complete onboarding (it auto-writes the ego-browser skill into ~/.claude/skills/ and migrates the user's login/cookies/profile — the source of the DataDome-trusted session), then /ego-browser + Full access. Re-run the check, then retry Round 2a. Do not substitute a headless browser — OpenEvidence sits behind DataDome-class protection and automated browsers get blocked; ego passes because it reuses the user's trusted session. Note: if a oe-<id>.dump.json already exists, Round 2b (transform) still runs without ego — offer that.

Verified facts (don't re-discover)

• Ask box: textarea[placeholder="Ask a medical question..."]. Follow-up box: textarea[placeholder="Ask a follow-up question..."]. Submit with Enter.
• Every turn mints a NEW /ask/<uuid> URL rendering the whole thread up to that turn — export the latest id.
• Flaky first paint → dump.mjs reloads before parsing.
• Public = ask URL once set: Share → radio input[value="ANYONE_WITH_LINK"] → Done.
• Per-turn refs: parse via an ordered token stream (compareDocumentPosition); a turn's numbered refs are only the external anchors AFTER its References heading (the one before it is the figure source link — excluding it keeps citation numbers aligned; refCount === max citation number). Citations restart at [1] each turn.
• Crossref: api.crossref.org/works/{doi}/transform/application/vnd.citationstyles.csl+json. Set env CROSSREF_MAILTO to join the polite pool (optional). PMID→DOI via NCBI esummary. CSL-JSON title is a string; container names may contain <scp> etc. (strip tags).
• ego Node runtime ignores shell env → pass inputs via scripts/params.json (a file). oe-transform.mjs instead takes the dump path as argv[2] (it's plain Node).
• ESM: both scripts use import/await import('node:fs'), not require. Absolute paths (cwd is / under ego).

Inputs

• New thread: QUESTION. Follow-up: FOLLOWUP + existing ARTICLE_ID.
• QUESTIONS — every turn's question, in order (card titles). New: [QUESTION]. Follow-up: [...prior, FOLLOWUP]. Unknown earlier turns fall back to Turn N.
• OUT_DIR — default $HOME/oe-answers (omit it in params.json to use the default).

Round 1 — new thread (ego): submit the question

ego-browser nodejs <<'EOF'
const task = await useOrCreateTaskSpace('openevidence ask')
await openOrReuseTab('https://www.openevidence.com/', { wait: true, timeout: 40 })
await waitForLoad(); await wait(2)
const QUESTION = `<<<QUESTION>>>`
await click('textarea[placeholder="Ask a medical question..."]', { label: 'focus ask box' })
await fillInput('textarea[placeholder="Ask a medical question..."]', QUESTION)
await wait(0.5); await pressKey('Enter'); await wait(4)
const m = ((await pageInfo()).url||'').match(/\/ask\/([0-9a-f-]{16,})/)
cliLog('ARTICLE_ID: ' + (m ? m[1] : 'NONE'))
EOF

Round F — follow-up (ego): append to a thread

ego-browser nodejs <<'EOF'
const task = await useOrCreateTaskSpace('openevidence ask')
const ARTICLE_ID = `<<<EXISTING THREAD ID>>>`
const FOLLOWUP   = `<<<FOLLOW-UP QUESTION>>>`
await gotoAndWait('https://www.openevidence.com/ask/' + ARTICLE_ID, { timeout: 45, settle: 3 }); await wait(2)
await click('textarea[placeholder="Ask a follow-up question..."]', { label: 'focus follow-up' })
await fillInput('textarea[placeholder="Ask a follow-up question..."]', FOLLOWUP)
await wait(0.5); await pressKey('Enter'); await wait(4)
const m = ((await pageInfo()).url||'').match(/\/ask\/([0-9a-f-]{16,})/)
cliLog('NEW_ARTICLE_ID: ' + (m ? m[1] : 'NONE'))
EOF

Round 2 — capture (ego) → transform (standalone)

2a. Write params, capture the DOM with ego:

SKILL="$HOME/.claude/skills/ask-oe-with-ego"
# OUT_DIR is optional — omit it and dump.mjs defaults to $HOME/oe-answers.
cat > "$SKILL/scripts/params.json" <<'JSON'
{
  "ARTICLE_ID": "<<<LATEST TURN ID>>>",
  "QUESTIONS": ["<<<Q1 text>>>", "<<<Q2 text>>>"]
}
JSON
ego-browser nodejs < "$SKILL/scripts/dump.mjs"     # → oe-<id>.dump.json  (prints the exact NEXT command)

2b. Transform standalone — no ego, no browser:

node "$SKILL/scripts/oe-transform.mjs" "$HOME/oe-answers/oe-<LATEST TURN ID>.dump.json"
# → oe-<id>.html (+.bib .csl.json .ama.txt)  |  crossref:N/M

Step 2b runs anywhere Node ≥18 is present, on the .dump.json alone — that's the standalone half. If ego is missing but a dump exists, run only 2b.

Storage (SQLite) — scripts/store.mjs

oe-transform.mjs auto-persists every dump into scripts/oe-store.db (built-in node:sqlite; best-effort, non-fatal if unavailable). This means you can re-render a thread later without re-capturing from OpenEvidence — no ego, no DataDome. The DB (and its -wal/-shm sidecars, plus transient params.json) is .gitignored.

SKILL="$HOME/.claude/skills/ask-oe-with-ego"
node "$SKILL/scripts/store.mjs" list                       # every stored thread (date, id, turns, refs, public, first Q)
node "$SKILL/scripts/store.mjs" get  <article_id> d.json   # export a stored dump back to JSON
node "$SKILL/scripts/oe-transform.mjs" d.json              # …then re-render offline from it
node "$SKILL/scripts/store.mjs" put  <dump.json>           # manual insert/upsert
node "$SKILL/scripts/store.mjs" rm   <article_id>          # delete

Optional — cloud backup to Turso (scripts/backup.mjs)

Entirely optional; nothing in the ask/transform/local-store flow depends on it. It mirrors the local SQLite store to a Turso (libSQL) cloud DB using the turso CLI the user already authed — no app tokens, no npm client. Only reach for it when the user asks to back up / sync to the cloud.

Preflight (skip the whole feature if this fails — do not block anything else):

command -v turso >/dev/null 2>&1 && turso auth whoami >/dev/null 2>&1 && echo "TURSO OK" || echo "TURSO OFF"

If TURSO OFF, just tell the user cloud backup is available once they install + log in — curl -sSfL https://get.tur.so/install.sh | bash then turso auth login — and carry on; local .db is the source of truth regardless.

SKILL="$HOME/.claude/skills/ask-oe-with-ego"
node "$SKILL/scripts/backup.mjs" backup    # local oe-store.db → cloud (creates db first time, else upserts)
node "$SKILL/scripts/backup.mjs" status    # cloud row count + URL
node "$SKILL/scripts/backup.mjs" restore   # cloud → local (INSERT OR REPLACE)

Cloud db name defaults to oe-store (override with env TURSO_DB). backup.mjs self-checks the CLI/auth and exits with install guidance if absent — it never throws into the rest of the skill.

After Round 2

• Report the public link and file paths; SendUserFile([htmlPath], { display:'render', status:'normal' }) to show the page, attach .bib/.csl.json/.ama.txt on request.
• Close ego in a final heredoc: await completeTaskSpace('openevidence ask', { keep: false }).

Adding more follow-ups

Repeat Round F on the latest id → Round 2 with QUESTIONS extended. Renderer and bibliography scale to any turn count.

Fallbacks

• EGO MISSING → capture can't run; hint install (see Preflight). Transform still works on an existing dump.
• oe-undefined.* / empty dump → params.json unread or ARTICLE_ID missing (env vars don't propagate — the file is the only channel).
• Answer never completes (dump poll exits low) → still streaming; rerun 2a.
• crossref:0 → Crossref/NCBI unreachable; entries fall back to OpenEvidence-text synthesis (valid, less rich). Check network from the transform host.
• Citations off by one → a pre-References inline anchor slipped into a turn's ref list; the inRefs gate must exclude it.
• Keep it private → delete the Share block in dump.mjs; HTML then shows Private.