Skip to content

Commit

Permalink
add use proxy and llm baseurl examples
Browse files Browse the repository at this point in the history
  • Loading branch information
datehoer committed Aug 27, 2024
1 parent 16f98ce commit 2ba70b9
Showing 1 changed file with 27 additions and 0 deletions.
27 changes: 27 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,33 @@ result = crawler.run(
print(result.extracted_content)
```

### Extract Structured Data from Web Pages With Proxy and BaseUrl

```python
from crawl4ai import WebCrawler
from crawl4ai.extraction_strategy import LLMExtractionStrategy

def create_crawler():
crawler = WebCrawler(verbose=True, proxy="http://127.0.0.1:7890")
crawler.warmup()
return crawler

crawler = create_crawler()

crawler.warmup()

result = crawler.run(
url="https://www.nbcnews.com/business",
extraction_strategy=LLMExtractionStrategy(
provider="openai/gpt-4o",
api_token="sk-",
base_url="https://api.openai.com/v1"
)
)

print(result.markdown)
```

## Documentation 📚

For detailed documentation, including installation instructions, advanced features, and API reference, visit our [Documentation Website](https://crawl4ai.com/mkdocs/).
Expand Down

0 comments on commit 2ba70b9

Please sign in to comment.