Smart Web Scraper

This API provides an intelligent extraction engine designed to turn websites into structured data. It supports batch processing (up to 10 URLs) and features an AI Analysis that reads, structures, and categorizes content automatically.

It handles the heavy lifting: resolving relative links, cleaning DOM disturbance like ads, & scripts, and formatting text for machine reading.

Features & Limitations

• Clean Extraction: Removes navbars, footers, and scripts to focus on the main content.
• AI Summarization: Optional AI analysis to extract key points and sentiment.
• SSRF Protection: Built-in security against internal network scanning.

Known Limitations

This endpoint uses a stable, standard browser signature. It will NOT work on sites requiring complex JS rendering or aggressive bot protection (e.g., Google, YouTube, Facebook, Twitter/X, or sites behind Cloudflare “Under Attack” mode).

Endpoint

POST /api/v1/web-search

Request Parameters

The following parameters are accepted in the request body:

Field	Type	Required	Description
url	string	array	Yes
summarize	boolean	Optional	Set to true to enable AI analysis. (Default: false).

Request Examples

cURL

curl -X POST https://heavstal.com.ng/api/v1/web-search \
-H "Content-Type: application/json" \
-H "x-api-key: <YOUR_API_KEY>" \
-d '{"url": ["https://techcrunch.com/article-1", "https://verge.com/article-2"], "summarize": true}'

JavaScript (Fetch)

const response = await fetch('https://heavstal.com.ng/api/v1/web-search', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'x-api-key': '<YOUR_API_KEY>'
  },
  body: JSON.stringify({
    url: [
      'https://techcrunch.com/article-1',
      'https://verge.com/article-2'
    ],
    summarize: true
  })
});

const res = await response.json();
console.log(res.data);

Heavstal SDK

import { search } from '@heavstal/api';

// true = summarize with AI
const article = await search.webSearch("https://example.com/blog", true);

console.log(article.content.summary);

Python

import requests

url = "https://heavstal.com.ng/api/v1/web-search"
headers = {
    "Content-Type": "application/json",
    "x-api-key": "<YOUR_API_KEY>"
}
payload = {
    "url": ["https://techcrunch.com/article-1"],
    "summarize": True
}

response = requests.post(url, headers=headers, json=payload)
print(response.json()["data"])

Response Examples

Success Response (Batch Mode)

{
  "status": "success",
  "creator": "HEAVSTAL TECH",
  "batch_size": 2,
  "data": [
    {
      "url": "https://techcrunch.com/article-1",
      "meta": { "title": "Startup raises $10M..." },
      "content": {
        "summary": "A startup focused on AI has raised Series A funding...",
        "key_points": ["$10M raised", "Led by VC Firm X", "Plans to hire 50 engineers"],
        "sentiment": "positive"
      }
    },
    {
      "url": "https://verge.com/article-2",
      "meta": { "title": "New Phone Review" },
      "content": { "summary": "..." }
    }
  ]
}

Error Response

{
  "status": "error",
  "error": "Failed to scrape URL. Target may be blocking automated requests."
}