LLM Scraper is a TypeScript library that allows you to convert any webpages into structured data using LLMs.
Tip
Under the hood, it uses function calling to convert pages to structured data. You can find more about this approach here
- Supports Local (GGUF), OpenAI, Groq chat models
- Schemas defined with Zod
- Full type-safety with TypeScript
- Based on Playwright framework
- Streaming objects
- Supports 4 input modes:
html
for loading raw HTMLmarkdown
for loading markdowntext
for loading extracted text (using Readability.js)image
for loading a screenshot (multi-modal only)
Make sure to give it a star!
-
Install the required dependencies from npm:
npm i zod playwright llm-scraper
-
Initialize your LLM:
OpenAI
npm i @ai-sdk/openai
import { openai } from '@ai-sdk/openai' const llm = openai.chat('gpt-4o')
Groq
npm i @ai-sdk/openai
import { createOpenAI } from '@ai-sdk/openai' const groq = new OpenAI({ baseURL: 'https://api.groq.com/openai/v1', apiKey: process.env.GROQ_API_KEY, }) const llm = groq('llama3-8b-8192')
Local
import { LlamaModel } from 'node-llama-cpp' const llm = new LlamaModel({ modelPath: 'model.gguf' })
-
Create a new scraper instance provided with the llm:
import LLMScraper from 'llm-scraper' const scraper = new LLMScraper(llm)
In this example, we're extracting top stories from HackerNews:
import { chromium } from 'playwright'
import { z } from 'zod'
import { openai } from '@ai-sdk/openai'
import LLMScraper from 'llm-scraper'
// Launch a browser instance
const browser = await chromium.launch()
// Initialize LLM provider
const llm = openai.chat('gpt-4o')
// Create a new LLMScraper
const scraper = new LLMScraper(llm)
// Open new page
const page = await browser.newPage()
await page.goto('https://news.ycombinator.com')
// Define schema to extract contents into
const schema = z.object({
top: z
.array(
z.object({
title: z.string(),
points: z.number(),
by: z.string(),
commentsURL: z.string(),
})
)
.length(5)
.describe('Top 5 stories on Hacker News'),
})
// Run the scraper
const { data } = await scraper.run(page, {
schema,
mode: 'html',
})
// Show the result from LLM
console.log(data?.top)
await page.close()
await browser.close()
Replace your run
function with stream
to get a partial object stream (Vercel AI SDK only).
// Run the scraper
const { stream } = await scraper.stream(page, {
schema,
mode: 'html',
})
// Stream the result from LLM
for await (const data of stream) {
console.log(data.top)
}
As an open-source project, we welcome contributions from the community. If you are experiencing any bugs or want to add some improvements, please feel free to open an issue or pull request.