baihe26/zju-literature-downloader

ZJU Literature Downloader

This skill turns the verified workflow into a repeatable, legally scoped process for finding, downloading, and reading papers through the user's Zhejiang University Library / WebVPN access.

Boundaries

Use only the user's legitimate institutional access. Do not bypass paywalls, DRM, CAPTCHA, Cloudflare, publisher bot checks, or two-factor authentication. If a page asks for CAPTCHA, QR login, SMS/OTP, Cloudflare, publisher bot checks, or a security challenge, stop and ask the user to complete it in Chrome.

Avoid mass downloading. Work in small batches, preferably after the user confirms the paper list. Leave a clear audit trail of what was downloaded, from where, and whether supporting information was found.

Do not ask the user to paste institutional passwords, CAS credentials, OTP codes, recovery codes, or session tokens into chat or terminal. If the user offers a password, decline and use the handoff-login workflow instead.

Exception for ZJU CAS saved-login pages: if the user explicitly says that Chrome has already filled the ZJU CAS credentials and authorizes clicking the login/confirm button, the agent may click that button once on the ZJU CAS/WebVPN/institutional SSO page without reading, copying, or typing any credential. This exception does not apply to CAPTCHA, QR login, SMS/OTP, publisher bot checks, or any page outside the expected institutional login flow.

Do not inspect or export cookies, passwords, local storage, browser profiles, or session files. Use the browser's already-authenticated page context only.

Preconditions

Before attempting downloads, confirm these conditions:

1. Chrome is open on the user's machine.
2. The user has personally logged in to Zhejiang University Library / WebVPN in Chrome.
   • Common pages: https://libweb.zju.edu.cn/, https://webvpn.zju.edu.cn/, https://zju.summon.serialssolutions.com/.
3. Chrome remote debugging is allowed for the current browser instance.
   • Ask the user to open chrome://inspect/#remote-debugging.
   • They must enable Allow remote debugging for this browser instance.
4. The environment can run Node.js 22+.
   • Try node --version.
   • If node is not on PATH in Codex Desktop, try %LOCALAPPDATA%\OpenAI\Codex\bin\node.exe.
5. The web-access CDP proxy is available or can be started.
   • Typical Claude Code path: %USERPROFILE%\.claude\skills\web-access-main\scripts\check-deps.mjs.
   • Typical shared agent path: %USERPROFILE%\.agents\skills\web-access-main\scripts\check-deps.mjs.
   • In Codex-only setups also check %USERPROFILE%\.codex\skills\web-access-main\scripts\check-deps.mjs.
6. The user has approved the target output folder.

If Claude Code says this skill is not installed, install or copy it to:

$env:USERPROFILE\.claude\skills\zju-literature-downloader

Codex and other agent setups may instead use .codex\skills or .agents\skills; treat the three locations as install targets, not as different skill versions.

Batch Scope

Small batches are supported when the user provides a definite DOI/title/PMID list.

Recommended limits:

• normal batch: 5-10 papers
• upper practical batch: 15-20 papers, with pauses and a manifest
• stop immediately if publisher checks, CAPTCHA, WebVPN expiry, or unusual download prompts appear

Do not turn a broad keyword search into unlimited automatic downloading. Do not download whole journal issues, volumes, or large result sets.

Status Categories

Classify every paper into one of these statuses, and keep the status in the manifest:

downloaded
downloaded_with_si
si_only_downloaded
cas_waiting_user
institutional_login_waiting_user
cas_resolved_retry_needed
publisher_verification_waiting_user
sciencedirect_robot_check
retry_after_user_verification
do_not_auto_retry
url_needs_repair
summon_no_link
summon_unreachable
browser_pdf_viewer_fallback_needed
publisher_blocked_waiting_user
no_authorized_pdf_found
failed_after_retry
ip_not_authorized
purchase_required

Use cas_waiting_user when the browser is visibly at Zhejiang University CAS / unified identity authentication. Use institutional_login_waiting_user when the paper is still at an institution selector, CARSI/Shibboleth/OpenAthens handoff, or equivalent SSO step before CAS. Both are non-final states; keep cas_waiting_user for backward-compatible manifests.

Use si_only_downloaded when one or more official supporting-information files were verified but the main article PDF remains unavailable through the user's authorized access. Do not call this downloaded.

Use summon_unreachable when the Summon SPA repeatedly returns ERR_EMPTY_RESPONSE or cannot load after a single fresh retry. Use browser_pdf_viewer_fallback_needed when Chrome visibly displays the PDF but browser-context fetch() cannot retrieve its bytes. Neither status means the paper is unavailable; both specify the next legitimate fallback.

Use publisher_verification_waiting_user or sciencedirect_robot_check when a publisher page shows "Are you a robot?", CAPTCHA, Cloudflare, bot verification, or another anti-automation challenge. Do not treat this as a final failure, but do not try to solve it automatically.

Start Chrome Control

Use the web-access CDP proxy when available.

On Windows PowerShell:

$node = "node"
if (-not (Get-Command node -ErrorAction SilentlyContinue)) {
  $node = "$env:LOCALAPPDATA\OpenAI\Codex\bin\node.exe"
}
$checkDepsCandidates = @(
  "$env:USERPROFILE\.claude\skills\web-access-main\scripts\check-deps.mjs",
  "$env:USERPROFILE\.agents\skills\web-access-main\scripts\check-deps.mjs",
  "$env:USERPROFILE\.codex\skills\web-access-main\scripts\check-deps.mjs"
)
$checkDeps = $checkDepsCandidates | Where-Object { Test-Path $_ } | Select-Object -First 1
if (-not $checkDeps) { throw "web-access-main/scripts/check-deps.mjs not found" }
& $node $checkDeps

Then test:

Invoke-WebRequest -UseBasicParsing -Uri "http://127.0.0.1:3456/targets" -TimeoutSec 10

If this hangs or fails:

• Ask the user to confirm the remote debugging checkbox.
• Check %TEMP%\cdp-proxy.log.
• Do not attempt to read Chrome session files.

Core CDP Proxy API

The usual web-access proxy is http://127.0.0.1:3456.

Endpoint	Method	Purpose
/health	GET	Proxy health and Chrome connection status
/targets	GET	List open Chrome tabs
/new?url=...	GET	Create a new background tab
/navigate?target=...&url=...	GET	Navigate an existing tab
/close?target=...	GET	Close a tab
/info?target=...	GET	Get page title, URL, and readyState
/eval?target=...	POST body=JS	Execute JavaScript in a tab and return { value: ... }
/clickAt?target=...	POST body=CSS selector	Click the center of a visible element

Use /navigate rather than /new for Summon URLs containing #! fragments. If a #! fragment is stripped, Chrome may open about:blank or a wrong page. If curl is unavailable, use PowerShell Invoke-WebRequest for simple checks or Node.js fetch() for proxy calls.

Recommended Search Workflow

Prefer the library discovery route before direct publisher pages. It is more stable and less likely to trigger bot protection.

1. Search by DOI or exact title in ZJU Summon:
   • https://zju.summon.serialssolutions.com/search?#!/search?pn=1&ho=t&include.ft.matches=f&l=en&q=<URL-encoded DOI or title>
2. Open Summon URLs through scripts/cdp_open_url.mjs or otherwise fully URL-encode the nested URL when calling the CDP proxy. Summon uses #!; if the # fragment is stripped, Chrome may open about:blank or the wrong page.
3. Read the result page with /eval.
4. Extract links whose visible text or aria-label is:
   • PDF
   • 在线全文
   • Full Text
   • View PDF
   • publisher-specific full-text entries.
5. Prefer the result's PDF link when present.
6. Open the PDF link in a new background tab through the CDP proxy.
7. If the publisher shows a security challenge, ask the user to complete it manually.
8. Once the PDF is visible in Chrome, use scripts/browser_pdf_downloader.mjs to save it from the authenticated browser context.

Summon Field Notes

When extracting PDF, online full text, or publisher links from Summon:

1. Skip empty href values. The first online-full-text match can be a facet toggle rather than a real link.
2. Treat Summon as a slow SPA: wait about 6 seconds after navigation, then retry link extraction up to 3 times with short pauses.
3. If Summon redirects to login.serialssolutions.com/sso/login and says the IP is outside the authorized range, record ip_not_authorized. In that case the user may need the ZJU RVPN/SSLVPN client, not only WebVPN in a browser page, because CDP-controlled Chrome tabs may still use the local network IP.
4. If Summon has no PDF but has an online-full-text link, follow that route before constructing a publisher URL manually.
5. If the page consistently shows ERR_EMPTY_RESPONSE or fails to render after one fresh navigation and the normal SPA waits, record summon_unreachable. Do not loop on Summon: try the DOI's publisher landing page or a confirmed OA route once, still using the user's authorized browser session where required.

Publisher-Specific Patterns

Use these as DOI-based repair hints only after confirming the paper identity. Do not claim access if the page or PDF is not actually verified.

ACS Publications

For DOI prefix 10.1021/...:

https://pubs.acs.org/doi/pdf/<doi>

ACS supporting information often follows:

https://pubs.acs.org/doi/suppl/<doi>/suppl_file/<journal-code>_si_001.pdf

The verified example 10.1021/acs.biomac.4c00102 used bm4c00102_si_001.pdf.

Start from the ACS article landing page whenever possible. A short Chinese 请稍候... interstitial can clear by itself; wait about 5-10 seconds and inspect the page once before classifying it as a publisher challenge. A readable abstract page does not prove that the main PDF is authorized: if it still shows Available to Purchase or Access through Institution, record the appropriate institutional-login or purchase status and leave any sign-in selection for the user.

ACS supporting-information links shown on the article page may legitimately lead to ACS Figshare / ndownloader.figstatic.com downloads even when the main PDF is not authorized. Follow visible official links, verify the resulting SI PDF, and record si_only_downloaded until the main article is also verified. Do not guess a supplement filename beyond an article-page link, and do not treat accessible SI as permission to retrieve the paywalled main paper.

Wiley

For DOI prefixes such as 10.1002/... or 10.1111/..., first navigate to the authenticated Wiley article page, then fetch from that page's own origin:

<location.origin>/doi/pdfdirect/<doi>?download=true

Do not hardcode onlinelibrary.wiley.com: Summon may authenticate through a subdomain such as advanced.onlinelibrary.wiley.com, and cross-origin fetches can fail. Also avoid navigating the tab directly to pdfdirect, because the browser may start a download and leave the page context unusable. Prefer page-context fetch().

Springer Nature

For DOI prefixes such as 10.1007/... and 10.1186/...:

https://link.springer.com/content/pdf/<doi>.pdf

This usually works after the institutional login is resolved. Use browser-context fetch if direct shell download returns 401, 403, or an HTML login page.

Nature Communications

For OA DOI patterns such as 10.1038/s41467-..., try:

https://www.nature.com/articles/<article-id>.pdf

Extract <article-id> from the DOI suffix and still verify the PDF header and text.

bioRxiv

For DOI prefix 10.1101/..., try:

https://www.biorxiv.org/content/<doi>v1.full.pdf

If version 1 returns 404, try the article page or a later version. Verify that the title matches.

Frontiers

For DOI prefix 10.3389/..., try:

https://www.frontiersin.org/articles/<doi>/pdf

If it fails, open the article page first and use its visible PDF link.

RSC Publishing

For DOI prefix 10.1039/..., do not assume ZJU has authorized full-text access. If PDF links return 404 or purchase pages, inspect the article page and record no_authorized_pdf_found or purchase_required rather than retrying aggressively.

Publisher Verification and ScienceDirect

ScienceDirect and some publisher platforms may show "Are you a robot?", CAPTCHA, Cloudflare, bot verification, or similar checks after repeated direct DOI navigation or automated tab opening. These pages are security and anti-automation challenges, not ordinary login confirmations.

Reduce the chance of triggering them by using a conservative access pattern:

1. Prefer ZJU Summon / WebVPN / library 在线全文 links before direct doi.org -> publisher navigation.
2. Process ScienceDirect and other sensitive publishers one article at a time.
3. Keep a visible audit trail in the manifest; do not open many publisher tabs in parallel.
4. Wait for each page to settle before looking for Download PDF, View PDF, or PDF.
5. Reuse the same tab after the user completes a verification step instead of opening repeated new tabs.
6. Avoid retry loops. One failed automatic attempt is enough before handing the page to the user.

When a publisher verification page appears:

1. Stop automated actions on that tab.
2. Record the paper in publisher_verification.tsv or the main manifest with status publisher_verification_waiting_user; use sciencedirect_robot_check for ScienceDirect's "Are you a robot?" page.
3. Tell the user which paper and tab need manual attention.
4. Do not click CAPTCHA, Cloudflare, "Are you a robot?", bot-check, or similar challenge controls automatically.
5. After the user says the verification is complete, re-evaluate the tab and continue the download workflow.