Loader resource

The loader: resource reads a file, URL, or directory into structured Document objects - plain text plus metadata - and optionally splits them into chunks. It is the ingestion step of a RAG pipeline: load, then feed the output into vectorStore: or embedding: for indexing.

Where it runs

Both workflow mode and agent mode. In agent mode, the same loader is available as the load_document built-in tool.

Basic usage

# resources/load.yaml
actionId: load
name: Load Report
loader:
  type: pdf
  source: /data/quarterly-report.pdf

# resources/loadAndChunk.yaml
actionId: loadAndChunk
name: Load and Chunk for RAG
loader:
  type: text
  source: /data/handbook.txt
  chunkSize: 500
  chunkOverlap: 50
  chunkSplitter: recursive

Configuration options

Option	Description
type	Loader to use: text (default), pdf, html, csv, directory
source	File path, URL (for html), or directory path (required)
columns	CSV only - optional column filter (empty = all columns)
password	PDF only - decryption password for encrypted PDFs
chunkSize	When set (> 0), splits each loaded document into chunks of this size
chunkOverlap	Characters of overlap between consecutive chunks
chunkSplitter	Splitting strategy: recursive (default), token, or markdown

Loader types

type	Reads
text	Plain text file
pdf	PDF file, page text extracted
html	A URL - fetches and extracts readable text
csv	CSV file, one Document per row (or per columns filter)
directory	Every file in a directory, one Document per file

Output

{
  "documents": [
    { "content": "...", "metadata": { "source": "/data/handbook.txt", "chunk": 0 } }
  ],
  "count": 3
}

documents is directly usable as the documents: input to a vectorStore: add_documents operation - each content/metadata pair maps onto a VectorStoreDocument.

Size limit

Every loader type reads its source fully into memory before returning - fine for a report or a handbook, but a workflow doing bulk ingestion (a large CSV, a directory of documents) risks an OS-level, no-core-dump SIGKILL once that content gets copied again downstream (chunkSize splitting, json.Marshal in the result, a vectorStore: step embedding it). Any single file over 256 MiB is rejected with a clear error instead - type: directory/notion skip an oversized file and keep loading the rest (a batch load should not abort over one large file); every other type fails that one load outright, the same way an explicit --file/stdin over its own 256 MiB default already does (see Input sources).

Raise or lower it with KDEPS_LOADER_MAX_BYTES (bytes; 0 disables the check):

KDEPS_LOADER_MAX_BYTES=1073741824 kdeps run workflow.yaml  # 1 GiB, for a workflow that loads bulk data

RAG pipeline example

# resources/load.yaml
actionId: load
loader:
  type: pdf
  source: "{{ get('filePath') }}"
  chunkSize: 500
  chunkOverlap: 50

# resources/index.yaml
actionId: index
requires: [load]
vectorStore:
  operation: add_documents
  provider: qdrant
  url: http://localhost:6333
  collection: docs
  documents: "{{ get('load').documents }}"
  embedModel: text-embedding-3-small

See also

• Vector store resource - index and search the loaded documents
• Embedding resource - lightweight keyword-based alternative for local, API-key-free RAG
• Resources overview - all resource types