# PageSift 1.2.2 — full reference for LLMs and agents

> The web and document layer for AI agents: any URL, document, email, archive or source file in, clean Markdown or JSON out. One self-hosted Rust binary with a CLI, Firecrawl and Jina Reader APIs, an MCP server, and packages for Python, Node.js, Go, Java, C#, Ruby, PHP and more.

Website: https://pagesift.tech
Repository: https://github.com/YOLOW92/pagesift
License: AGPL-3.0, or a commercial license (https://github.com/YOLOW92/pagesift/blob/main/docs/COMMERCIAL.md)

## What it is

pagesift reads the web and documents for software that thinks: it fetches a URL, rendering JavaScript only when that adds content, or opens a local file, and returns the content as Markdown or structured JSON. It covers single pages, whole sites, scanned PDFs, browser actions, change monitoring and research runs driven by your own model.

Can replace or sit alongside: Firecrawl (same v1/v2 API, self-hosted), Jina Reader (`/r/` and `/s/`), Crawl4AI, trafilatura, Readability, pymupdf4llm, Docling, Kreuzberg, xberg.

## Install

| Surface | Install | First call |
|---|---|---|
| CLI | `cargo install pagesift --locked` | `pagesift scrape https://example.com` |
| Python | `pip install pagesift` | `pagesift.scrape("https://example.com").markdown` |
| Node.js | `npm install @pagesift/node` | `(await new Scraper().scrape("https://example.com")).markdown` |
| Server | `docker run -p 3002:3002 ghcr.io/yolow92/pagesift:1.2.2` | `curl localhost:3002/r/https://example.com` |
| MCP | — | `claude mcp add pagesift -- pagesift mcp` |

Platforms: Linux x86_64 / aarch64 (glibc ≥ 2.28 and musl), macOS arm64, Windows x86_64; Docker image `ghcr.io/yolow92/pagesift`.
Runtime: nothing to install for HTTP, PDF and Office. Optional: a browser for JavaScript pages, an NVIDIA GPU for OCR, your own model for `/v2/agent`.

## CLI

```bash
pagesift scrape https://example.com                          # Markdown to stdout
pagesift scrape report.pdf                                   # local PDF, Office, e-book, email, archive, image, code or WARC
pagesift --ocr scrape scan.pdf                               # OCR scanned pages; models download on first use
pagesift crawl https://docs.rs --limit 500 --out ./data/     # one file per page
pagesift crawl https://www.example.edu --query "graduate admission" --adaptive
pagesift extract --schema schema.json https://example.com    # JSON by CSS selectors, repaired when the page changes
pagesift doctor                                              # check browsers, GPU and models
```

## HTTP APIs

Firecrawl v1/v2 and Jina Reader endpoints on the same self-hosted binary, plus a file-upload API:

```bash
docker run -d -p 3002:3002 -v pagesift-data:/data -e PAGESIFT_API_KEYS=sk-test ghcr.io/yolow92/pagesift:1.2.2

curl -s localhost:3002/v2/scrape -H 'Authorization: Bearer sk-test' -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"]}'                        # Firecrawl API
curl -s localhost:3002/r/https://example.com -H 'Authorization: Bearer sk-test'    # Jina Reader
curl -s localhost:3002/v2/parse -H 'Authorization: Bearer sk-test' \
  -F file=@report.pdf -F 'options={"formats":["markdown"]}'                        # file upload
```

Firecrawl SDKs work by changing the API URL.

## MCP server

Start it for one agent over stdio, or as one shared local service over HTTP:

```bash
pagesift mcp                                                  # stdio, for one agent
pagesift mcp http --port 3003                                 # one shared local service
claude mcp add --scope user --transport http pagesift http://127.0.0.1:3003/mcp
```

Tools:

- `pagesift_scrape` — fetch one URL or file and return clean Markdown or structured JSON
- `pagesift_batch_scrape` — scrape many URLs as one persistent job
- `pagesift_crawl` — crawl a whole site as a job (sitemaps, robots.txt, per-host limits)
- `pagesift_poll_job` — poll a crawl or batch job for status and results
- `pagesift_cancel_job` — cancel a running job
- `pagesift_map` — discover the URLs of a site before fetching them
- `pagesift_interact` — drive a kept browser session: clicks, typing, scrolling
- `pagesift_interact_stop` — end a browser session
- `pagesift_agent` — research run that answers a question from the web with your own model (only with a configured model)
- `pagesift_agent_status` — status of an agent research run (only with a configured model)

## Capabilities

- Clean content: main-content detection and Markdown that keeps headings, lists, tables, code and math
- Documents: own Rust readers for PDF, current and legacy Office, OpenDocument, e-books, email, archives, images, databases and text formats, including damaged files and CJK fonts
- OCR: scanned PDF pages through PP-OCRv6 and display formulas through PP-FormulaNet, in pure Rust on CPU or NVIDIA GPU
- Structured extraction: JSON by CSS selectors that are relocated when a site changes, or by a schema your model writes once and pagesift caches
- Browser actions: clicks, typing and scrolling on the page, plus sessions you can keep working in
- Agent: `POST /v2/agent` answers a question from the web with your own model, and checks each field against the page it came from
- Monitoring: scheduled checks that separate real changes from dates, counters and ads, with webhooks
- Crawling: sitemaps, query-focused crawls that stop when pages stop helping, trap handling, robots.txt and per-host limits
- Archives: WARC output on every crawl, and WARC files as input with no network
- Operations: multi-server gateway, workers and standby; job persistence, durable webhooks, metrics, audit log
- Safety: private-network and DNS-rebinding guards, and every incomplete result reports a warning instead of looking complete

## Inputs and outputs

Input: URLs, HTML, PDF including scanned pages (OCR), image files inside documents (OCR), Word (.docx, .doc), Excel (.xlsx, .xlsb, .xls), PowerPoint (.pptx, .ppt), OpenDocument, RTF, WordPerfect, HWP, EPUB, MOBI, FB2, email (.eml, .mbox, .msg), archives (zip, tar, gz, bz2, xz, 7z), SQLite, dBASE, CSV, Markdown and other text formats, 373 source-code and config languages, WARC archives.

Output: Markdown, HTML, raw HTML, links, images, metadata, schema-based JSON; warnings when a result is incomplete.

## Measured performance

Strongest result per domain from the published benchmark report (PageSift 1.1, xberg 1.3.9 measured 2026-11-04; full methods in the report):

- PDF → Markdown: 2.4 ms/page, 20.49× faster than pypdf (50.0 ms/page); resident server 176.5 files/s, 1.75× pdf_oxide
- HTML → Markdown: 0.24 ms/page, 8.12× faster than Turndown (1.9 ms/page), fidelity F1 1.000
- Main content: 1.0 ms/page (p50), 7.26× faster than crw (7.4 ms/page)
- Publication-date search: 0.31 ms/page, 6.45× faster than htmldate (fast)
- Office / e-books: 91.7 files/s, 3.40× xberg (27.0 files/s); structure kept 0.988
- Wild crawl coverage (polite, 40 sites × 60 pages): 0.740 vs Crawl4AI 0.529
- Peak memory per worker: PDF 32 MB vs Docling (CPU) 3,675 MB (113.1× less); HTML main content 20 MB vs Readability + Turndown 1,200 MB (60.8× less); crawl 218 MB vs spider 5,500 MB (25.2× less); 18–377 MB per worker across the 1.1 benchmarks without a browser

## Access-barrier recovery (itna, new in 1.2)

Opt-in with `[itna] enabled`, off by default. pagesift-itna classifies barriers, allows one bounded browser fallback, and shares one network budget across redirects and browser sub-requests. Sessions are leased per owner, origin, egress and fingerprint; HTTP cookies reach the browser only through an exact-origin handoff. Rate limits (429) and login walls never escalate. Interactive CAPTCHAs are reported as needing a human and are not solved.

## What's new in 1.2.2

Crawl, batch and extract jobs are restricted to their owner. Search and extract page attempts count against the daily quota, and cached pages are checked against robots.txt. Unsupported snapshot and base64-image options are now rejected instead of silently ignored.

## When to use it, and when not

Good fit: feeding web pages and documents to an LLM, a RAG index or an agent; self-hosting a Firecrawl- or Jina Reader-style service with no per-page fees; mixed input where the same call handles HTML, PDF, scans, Office files and e-books; small machines and containers, where memory is bounded by the cgroup limit.

Pick something else when: your input is standalone image files (OCR covers scanned pages inside PDFs, not standalone images); AGPL does not work for you and a commercial license is not an option; you need a managed cloud service — pagesift is self-hosted.

## Links

- Website: https://pagesift.tech
- Machine-readable summary: https://pagesift.tech/llms.txt
- Repository: https://github.com/YOLOW92/pagesift
- README: https://github.com/YOLOW92/pagesift/blob/main/README.md
- Architecture: https://github.com/YOLOW92/pagesift/blob/main/docs/architecture.md
- Benchmarks: https://github.com/YOLOW92/pagesift/blob/main/docs/benchmarks.md
- Release notes 1.2.2: https://github.com/YOLOW92/pagesift/blob/main/docs/releases/1.2.2.md
- Changelog: https://github.com/YOLOW92/pagesift/blob/main/CHANGELOG.md
- crates.io: https://crates.io/crates/pagesift
- PyPI: https://pypi.org/project/pagesift/
- npm: https://www.npmjs.com/package/@pagesift/node
- MCP listing on Glama: https://glama.ai/mcp/servers/YOLOW92/pagesift
- Community: Discord https://discord.gg/esA3wMZVvA · Issues https://github.com/YOLOW92/pagesift/issues · Discussions https://github.com/YOLOW92/pagesift/discussions