Scrape a web page and summarize it

Build an API that takes a URL, fetches the page with the scraper component, sends the extracted text to a local LLM, and returns a short summary as JSON.

The scraper component fetches a URL and returns its readable text - an httpClient GET, no browser, no external service. For pages that need JavaScript rendering, use the browser component instead.

Needs: Network access.

Step 1: create the project

mkdir web-scraper
cd web-scraper
mkdir resources

Step 2: define the route

Create workflow.yaml:

# workflow.yaml
apiVersion: kdeps.io/v1
kind: Workflow

metadata:
  name: web-scraper
  version: "1.0.0"
  targetActionId: response

settings:
  apiServer:
    portNum: 16402
    routes:
      - path: /summarize
        methods: [POST]
  agentSettings:
    pythonVersion: "3.12"

Step 3: fetch the page

Create resources/fetch.yaml:

# resources/fetch.yaml
actionId: fetch
name: Fetch page
validations:
  methods: [POST]
  routes: [/summarize]
  check:
    - get('url') != ''
  error:
    code: 400
    message: "url is required"
component:
  name: scraper                            # YAML component from the components asset set
  with:
    url: "{{ get('url') }}"
    selector: "{{ get('selector', '') }}"  # optional CSS selector; empty = whole page
    timeout: 30s

output('fetch').content will hold the extracted text.

Step 4: summarize with the LLM

Create resources/summarize.yaml:

# resources/summarize.yaml
actionId: summarize
name: Summarize page
requires: [fetch]
chat:
  model: llama3.2:1b
  role: user
  prompt: |
    Summarize the following web page content in 3-5 sentences.

    URL: {{ get('url') }}
    Content:
    {{ output('fetch').content }}
  jsonResponse: true          # force a JSON object as the reply
  jsonResponseKeys:
    - summary                 # the object must contain a "summary" key
  timeout: 60s

Step 5: return the summary

Create resources/response.yaml:

# resources/response.yaml
actionId: response
name: API response
requires: [summarize]
apiResponse:
  success: true
  response:
    url: "{{ get('url') }}"
    summary: "{{ output('summarize').summary }}"

Step 6: validate and run

kdeps validate .
export KDEPS_API_AUTH_TOKEN=dev-token
kdeps run . --dev

curl -X POST http://localhost:16402/summarize \
  -H "Authorization: Bearer $KDEPS_API_AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://go.dev/doc/"}'

Response:

{
  "success": true,
  "data": {
    "url": "https://go.dev/doc/",
    "summary": "The page is the documentation hub for the Go language. It links to the tour, effective Go, and the standard library reference..."
  }
}

Next steps

• Scraper component - selectors, timeouts, output shape
• Browser component - JavaScript-rendered pages
• LLM resource - JSON mode, streaming, tools
• searchWeb - find pages to scrape