web-fetch

Fetch web pages as clean Markdown. The script requests Markdown directly when a site offers it; otherwise, it extracts the main content from HTML and converts it to Markdown.

This is adapted from alleneubank's web-fetch skill. This version uses @kreuzberg/html-to-markdown-wasm for HTML-to-Markdown conversion; it still uses linkedom to parse HTML and select the page content. The wasm binary is embedded in the bundle, so the resource runs on Linux and macOS with no platform-specific install step.

Requirements

• Bun to run the bundled script or build it from source.
• Network access to the page being fetched.

Usage

Run the bundled skill resource from the repository root; no dependency install is needed for this path:

bun skill/resources/fetch.js "https://example.com/article"

The extracted Markdown is printed to standard output, so it can also be saved to a file:

bun skill/resources/fetch.js "https://example.com/article" > article.md

For metadata from an HTML page, pass --meta after the URL:

bun skill/resources/fetch.js "https://example.com/article" --meta

--meta prints the page metadata (title/description from <head>, plus headings, links, and images from the extracted content) instead of the Markdown content. Markdown-native responses yield headings/links/images parsed from the Markdown text, without document metadata. Plain-text and JSON responses carry no extractable structure, so this mode produces an empty result for those.

To use this as an agent skill, install or link the skill/ directory in your agent's skills location and follow skill/SKILL.md. The resource is intended to run from inside the skill directory as bun ./resources/fetch.js "<url>".

How it works

1. Request the URL with Accept: text/markdown and return the response if its content type is Markdown. Plain-text and JSON responses are returned as-is.
2. Otherwise, request HTML if needed and parse it with linkedom.
3. Choose the longest matching article, main, [role="main"], .content, or #content element, falling back to the document body.
4. Remove common navigation and other non-content elements, then convert the remaining HTML with the embedded @kreuzberg/html-to-markdown wasm build.

This is a heuristic extractor, not a browser. Pages rendered only by JavaScript may have little or no usable content, and some sites may select the wrong region or omit content. Use a browser-based tool when the returned Markdown is incomplete.

Development

The editable source is src/fetch.ts; skill/resources/fetch.js is the bundled skill resource. To work from source:

cd src
bun install
bun fetch.ts "https://example.com/article"

After editing the source, rebuild the skill resource from src/ (this regenerates the vendored wasm blob, then bundles):

bun run build

src/vendor/html_to_markdown_wasm.js is a vendored copy of the wasm-bindgen loader; src/vendor/wasm_b64.ts is generated from the .wasm binary at build time and not committed. After bumping the wasm dependency, rebuild so both stay in sync.