PMC / OA Literature PDF Retrieval
Prerequisites
This skill is intended only for Open Access (OA) articles (CC BY / CC BY-NC / CC BY-NC-ND, etc.). Do not use it to bypass paywalls for subscription articles—that constitutes circumvention of copyright protection. Determine OA status: Unpaywall returns is_oa: true, or EuropePMC reports isOpenAccess: Y.
Institutional Subscription Scenario (Important Supplement)
When the user's institution/university has a subscription to the publisher's database, manual single-article downloads are fully compliant within the scope of authorization. However, note the following:
- On-campus network / Institutional IP range: Publishers recognize IP, not users. Degradation chain level 5 (publisher direct link) typically returns a 200 status directly, and Cloudflare will not intercept. In this scenario, the existing pipeline of this skill can obtain the subscription content without any login logic.
- Off-campus access: Prioritize the system-level VPN provided by the university (transparent to scripts, configure once); WebVPN (URL-rewriting proxy) is not supported due to varying rewriting rules and the need for session cookies.
- Do not write Shibboleth / OpenAthens auto-login: This requires local storage of unified identity authentication credentials, posing a security risk; also, IdP flows vary significantly between institutions, making a universal implementation infeasible.
- Batch download red line: Almost all publishers' institutional licenses distinguish between "manual single-article downloads" and systematic/bulk downloading. Even with a subscription, scripted bulk fetching violates license terms and can lead to the entire campus IP range being blocked. If you need large-scale full-text mining, apply for the publisher's TDM (Text Mining) API (e.g., Elsevier TDM API, Springer Nature Meta API) and use official quota channels, not this skill for batch loops.
Degradation Chain (Strict Order)
| Priority | Method | Command / Endpoint | Description |
|---|---|---|---|
| 1 | EuropePMC Structured Full Text | https://www.ebi.ac.uk/europepmc/webservices/rest/{PMCID}/fullTextXML | Most stable. Used to extract full-text structure precise to second-level headings. Returns JATS XML |
| 2 | Unpaywall Locate OA Version | https://api.unpaywall.org/v2/{DOI}[email protected] | Read oa_locations[].url_for_pdf, provides direct PDF link |
| 3 | PMC PoW Solving Download | scripts/pmc_pdf.py <PMCID> <out.pdf> | See below. Primary method for obtaining article PDFs |
| 4 | EuropePMC Supplementary Materials | https://www.ebi.ac.uk/europepmc/webservices/rest/{PMCID}/supplementaryFiles | Returns ZIP containing original figures (gr1…=Figure 1…)+ supplementary material PDFs (mmc1.pdf) |
| 5 | Publisher Direct Link | Various .../article-pdf/... links | Often returns 403 (Cloudflare). Try last |
Confirmed Dead Endpoints (Do Not Waste Time)
…/rest/{PMCID}/fullTextPDF→ 404 (Deprecated)…/rest/{PMCID}/fulltextRepo→ 403https://europepmc.org/articles/{PMCID}?pdf=render→ 403https://europepmc.org/backend/ptpmcrender.fcgi?accid=…&blobtype=pdf→ 403https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=…→ 404 (OA service migrated)- Article pages on
ashpublications.orgget stuck on Cloudflare's "Please wait…" challenge page in headless browsers, unusable
★ Prerequisite: PMC PDF Filenames are Not Fixed – Must Parse First
The filename in https://pmc.ncbi.nlm.nih.gov/articles/{PMCID}/pdf/<filename>.pdf varies by journal and year. Hardcoding main.pdf will result in a 404 in most cases.
| Tested Journal | Actual Filename |
|---|---|
| Am J Hematol | main.pdf |
| Blood Advances | main.pdf |
| HemaSphere | HEM3-10-e70338.pdf |
| Am J Hematol | AJH-101-1790.pdf |
| Liver International | LIV-46-0.pdf |
| Haematologica | 111300.pdf (Volume 111 + starting page 300, omits journal abbreviation) |
| Blood (NIH Author Manuscript) | nihms-2192199.pdf |
Parsing Methods (Priority order, tested: first method hits in all real cases)
- Article page
<meta name="citation_pdf_url" content="https://pmc.ncbi.nlm.nih.gov/articles/{PMCID}/pdf/XXX.pdf">← Most reliable grep -oE 'href="[^"]*?/pdf/[^"]+?\.pdf"'take first match- Fallback
pdf/main.pdf
Naming pattern reference (not reliable, for manual verification only): Wiley OA often <journal-abbrev>-<print-volume>-0.pdf (e.g., LIV-46-0.pdf); with an article number: <journal-abbrev>-<volume>-<article-number>.pdf (e.g., HEM3-10-e70338.pdf); same journal often <abbrev>-<volume>-<start-page>.pdf (e.g., AJH-101-1790.pdf); also variant "volume+start-page, no journal abbreviation" exists (e.g., 111300.pdf for Haematologica = Volume 111 + starting page 300).
When fetching the article page, be sure to discard cookies: Use a separate, cookie-less request to read the article page. The PDF request must still originate from a fresh cookie session. If cookies like
ncbi_sid/ncbi_pingergenerated from the article page are carried over to the PDF request, you will be escalated directly to reCAPTCHA (tested).scripts/pmc_pdf.pyimplements this.
Core Solution: Offline Solving of PMC Proof-of-Work
Principle
https://pmc.ncbi.nlm.nih.gov/articles/{PMCID}/pdf/<filename>.pdf switches between two challenge pages based on risk strategy:
- PoW Page (1817 bytes, title
Preparing to download ...) → Offline solvable, this is our desired channel; - reCAPTCHA Page (~20 KB, title contains
reCAPTCHA) → Cannot be solved offline.
Key to trigger PoW instead of reCAPTCHA: use a minimal UA, and do not access the article page before requesting the PDF.
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
Counterexamples (will be escalated to reCAPTCHA, tested):
- Full Chrome UA (
…AppleWebKit/537.36 … Chrome/1xx.0.0.0 Safari/537.36) - First
GETthe article page (carryingncbi_sid/ncbi_pinger) then request the PDF
PoW Algorithm (Reverse-engineered from pow-*.js, confirmed)
The PoW page embeds four constants: POW_CHALLENGE / POW_DIFFICULTY / POW_COOKIE_NAME / POW_COOKIE_PATH.
- Find the smallest non-negative integer
noncesuch thatsha256(challenge + str(nonce)).hexdigest()prefix equals'0' * difficulty; - Set the cookie:
<POW_COOKIE_NAME> = challenge + "," + nonce(pathPOW_COOKIE_PATH); - Re-request the PDF direct link with that cookie and
Referer: https://pmc.ncbi.nlm.nih.gov/articles/{PMCID}/→ Expect200status andContent-Type: application/pdf.
Typical difficulty = 4 (expected ~65536 hashes, completes locally in sub-seconds).
One-Click Script
# Recommended: auto-parse real PDF filename (via citation_pdf_url), then offline solve PoW
python3 scripts/pmc_pdf.py PMC13575452 ./out.pdf
# If you already parsed the filename, specify it directly to skip article page request
python3 scripts/pmc_pdf.py PMC13575452 ./out.pdf --pdf-name LIV-46-0.pdf
# Quiet mode (output results only)
python3 scripts/pmc_pdf.py PMC13575452 ./out.pdf --quiet
Script behavior: First, fetch the article page with minimal UA discarding cookies to parse the real PDF direct link → Then make a fresh cookie session cold request to that PDF → If direct PDF obtained, save it; otherwise parse PoW constants, solve, backfill cookie, and retry. To force a fresh challenge, just rerun (script does not persist cookies; each run is a fresh session).
★★ Two Hard Rules You Must Remember (Learned from Pain)
1. The second positional argument is the full output file path, not a directory. Passing a directory will raise
IsADirectoryError: [Errno 21] Is a directoryafter already obtaining the PDF, wasting the PoW solve and the PDF. ❌pmc_pdf.py PMC13482240 --outdir /path/to/dir✅pmc_pdf.py PMC13482240 /path/to/dir/xxx.pdf2. PMC's PDF channel has a "one shot" property. The first cold request almost always gets through PoW and yields a real PDF; a second request to the same PDF immediately after will almost certainly be escalated to reCAPTCHA (not just regular rate limiting where changing UA helps). Therefore, write the output path correctly the first time. Never "test run first to see." Tested: first round
status=200 type=application/pdf bytes=23970819, seconds later rerunning the same URL yieldstype=text/html bytes=21329(reCAPTCHA challenge page). Once this chance is wasted, jump directly to the "Publisher Website Degradation Route" in the next section; do not retry PMC repeatedly.
Troubleshooting
| Symptom | Cause | Action |
|---|---|---|
Returns HTML containing POW_CHALLENGE | Entered PoW channel (normal) | Let the script solve it |
Returns HTML containing recaptcha | Escalated to reCAPTCHA | Switch to minimal UA, clear session cookies, wait and retry; if still fails, use routes 4/5 |
| 404 / "Article not found" | PDF filename guessed incorrectly | Go back to the article page and parse citation_pdf_url; do not hardcode main.pdf |
nonce takes long to converge | difficulty increased | Print difficulty to assess; if >5, consider async/multiprocess solving |
| Solving succeeds but PDF returned is still HTML | Cookie domain/path mismatch | Confirm Cookie domain is pmc.ncbi.nlm.nih.gov, path is POW_COOKIE_PATH |
IsADirectoryError: Is a directory | Output argument treated as a directory | Second argument is the full output file path. The PDF is actually already obtained, but not saved—do not rerun immediately (will trigger reCAPTCHA); first fix the path, if already escalated, go directly to publisher website route |
| First success, immediate reCAPTCHA on second attempt | PMC's "one shot" rate limiting for the same PDF path | See hard rule 2 above; abandon PMC, use publisher website degradation route |
citation_pdf_url returns -mmc1.pdf | Article PDF not in repository, parsed supplementary material | Common for journals like Blood (JATS full text available but article PDF not in repository). Immediately abandon PMC route; do not substitute supplementary material for the article PDF (tested) |
★ IP-level rate limiting exists (tested) After repeated PDF requests to PMC in a short period (e.g., downloading multiple articles in succession), even with the correct minimal UA, you will be directly escalated to reCAPTCHA, and waiting 45 s does not recover. Presentation: "first success, then all failures." Action: ① For batch downloads, leave sufficient interval between requests; ② Once in reCAPTCHA state, do not retry repeatedly (only extends the ban); instead, complete other tasks first, or switch to route 4 (
supplementaryFiles) and publisher direct links; ③ If you indeed need multiple articles, prioritize on-demand download over batch preheat.
★★ Degradation Route: When PMC is Blocked by reCAPTCHA, Switch to Publisher Website (Tested Effective)
When two attempts at the PMC direct link are both escalated to reCAPTCHA (IP-level rate limiting that doesn't recover after waiting), and the target journal is a fully OA journal, going through the publisher website is more reliable than continuing to retry PMC:
UA="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"
# 1) DOI resolution → observe final publisher article page
curl -s -o /dev/null -w "final: %{url_effective}\nHTTP: %{http_code}\n" -A "$UA" -L "https://doi.org/<DOI>"
# 2) Parse real PDF direct link from article page
curl -s -A "$UA" "https://<publisher-article-page>" | grep -oiE '<meta name="citation_pdf_url"[^>]*>|href="[^"]*\.pdf[^"]*"' | sort -u | head
# 3) Fetch directly with Referer (publishers rarely check Referer, but it's safer to include)
curl -sL --retry 3 -A "$UA" -e "$ARTICLE_PAGE" -o "$OUT" "$PDF_URL"
★★ When Publisher Page is Completely Blocked by Cloudflare: Use Browser "Print to PDF" (Tested Effective)
ashpublications.org (ASH journals) currently exhibits complete Cloudflare blocking. The previous experience of "wait 10 s then open" could not be reproduced. A valid alternative path:
- Open the PMC full-text page (not PDF), bypassing the publisher site
- Use browser tools to print that page as PDF:
agent-browser pdf - When reCAPTCHA first pops up, executing open once more usually passes
Judgment point: If the target journal belongs to the ASH / ScienceDirect category with strong Cloudflare protection, do not retry curl repeatedly—low success rate and deepens bans; switch directly to a browser channel.
- Verified feasible: Haematologica / Ferrata Storti ——
doi.org/10.3324/haematol.2025.287698→haematologica.org/article/view/12187→haematologica.org/article/download/12187/78653→ 2.91 MB / 14 pages. - Not feasible: Publishers still blocked by Cloudflare-style challenges (ASH
ashpublications.org, Wileyonlinelibrary.wiley.com/doi/pdfdirect/..., De Gruyter, Thieme) — these can only fall back to PMC channel or a browser session. - Prejudgment: Only fully OA / Gold OA journals are worth trying this route (subscription journals will hit paywalls; bypassing them is prohibited).
★ Integrity Check: Interrupted Large File Download Can Cause xref Corruption
When curl fetching a PDF is interrupted by SIGTERM / timeout, the file header still reads %PDF-1.6, the size may seem reasonable, but pdfinfo will report Invalid XRef entry. Therefore, after download, do not rely solely on file header and size; all three must be satisfied:
head -c 8 "$OUT" # %PDF-…
tail -c 32 "$OUT" | od -c | tail -3 # Should see startxref … %%EOF
pdfinfo "$OUT" | grep -E "Pages|Page size" # Page count should match volume/issue/page estimates
Only when all three are complete is the file considered intact; otherwise, re-download fully (add --retry 3); do not submit a corrupted file.
Supplementary Materials and Figures
When the article PDF is blocked, the supplementaryFiles endpoint is often still usable, and the content is typically the accompanying figures and supplementary methods:
curl -sL -o sup.zip "https://www.ebi.ac.uk/europepmc/webservices/rest/PMC13049628/supplementaryFiles"
unzip -o sup.zip -d sup/
Naming convention (Wiley / ASH): *-gr1…*-grN correspond to Figure 1…N, *-ga1 is graphic abstract, *-mmc1.pdf is supplementary material.
When Using Browser Tools
Only use agent-browser when clicking, logging in, or rendering JS pages is necessary. For pure download scenarios, do not launch a browser—
PoW can be solved offline; launching Chromium only adds hundreds of MB of overhead and failure surface. If a browser is truly needed:
- Ensure a working node is added to PATH first (otherwise daemon startup times out);
- Be sure to
agent-browser closeafter task completion to avoid zombie daemons.
Verification Checklist
- Before downloading, confirm the target article is OA (or confirm currently within an institutional subscription IP range—in this case publisher direct links can be used directly but still only for manual single-article downloads; bulk is prohibited)
- Parse the real PDF filename from the article page's
citation_pdf_url(do not hardcodemain.pdf) - After saving, verify the file header is
%PDF - Check footer for
startxref/%%EOF(interrupted large files leave a head-normal, xref-corrupted half-file;pdfinfocan detect this) - Verify page count: Use
re.findall(rb'/Count\s+(\d+)', data)or count/Type /Pageinstances; do not trustfile out.pdfpage count (often misreports as "1 pages" on macOS); can also usemdls -name kMDItemNumberOfPages - Sample first and last pages (using
pdftotext -f 1 -l 1/-f N -l N) to confirm content is the article, not supplementary material or another article - Record
PMCID/DOI/ save path for future verification - If unsuccessful, honestly state the failure reason; do not pass off supplementary materials or figures as the article PDF