!

Legal Disclaimer

PipeAgent is a data distribution gateway. We do not own, verify, or endorse the data provided by third-party creators. Use at your own discretion.

Docs/provider / storage strategies

Feed Types & Storage Models

To maximize performance and minimize Token costs for AI Agents, PipeAgent uses a three-tier storage architecture. This ensures that Agents only process the data they actually need.

The Problem: Data Bloat & Token Burn

Traditional scraping returns raw HTML or massive unoptimized JSON. When an Agent reads a 100KB HTML page just to find one price, you are "burning" thousands of tokens on layout code rather than reasoning.

The Solution: 3 Specialized Models

PipeAgent providers choose a model during data ingestion (Push API).

1. Singleton (Atomic Mode)

  • Best for: Single JSON objects, configuration files, or real-time status updates (e.g., "Current Weather").
  • How it works: The entire payload is stored as a single blob. Each push atomically replaces the previous state.
  • Agent Payload: Small (1:1 with source).

    • 2. Collection (Batch Mode)

  • Best for: Lists that update as a group (e.g., "Top 50 Cryptos", "Current Stock Prices").
  • How it works: Providers push a batch of records. Consumers get a stable, paginated view of the latest version.
  • Agent Payload: Partial (Only requested items/fields).

    • 3. Stream (Append Mode)

  • Best for: Time-series data or growing lists (e.g., "Trade History", "News Feed").
  • How it works: Every push appends new records to the feed. Consumers navigate using time-based cursors.
  • Agent Payload: Optimized (Incremental updates).

    • ---

    Query Guardrails & Performance

    To ensure millisecond response times and multi-tenant isolation, PipeAgent enforces strict performance guardrails on all data queries.

    1. SQL-Like Filtering (Strict eq)

    You can filter data using standard query parameters, but only the Equality (eq) operator is supported for JSONB top-level fields inside the payload.

  • Allowed: GET https://api.pipeagent.dev/api/v1/feed/{id}?status=eq.bullish
  • Disallowed: Range (lt, gt) or pattern (ilike) filters on data fields are disabled to prevent slow database scans.

    • 2. JSONPath Projection

    Use JSONPath to extract only the specific fields your Agent needs.

  • Supported: Simple path extractions (e.g., $.metadata.price, $.items[0].price).
  • Disallowed: Filter expressions (e.g., $[?(@.price < 100)]) are disabled at the API level to prevent high CPU overhead. Only projection (path extraction) is allowed.

    • 3. Enforced Pagination

    To protect the context window of AI Agents, we strictly limit the amount of data returned in a single request.

  • Default Limit: 20 records.
  • Maximum Limit: 50 records. Any request for more than 50 records will be capped at 50.
  • Applies to: list reads, cursor reads, and /details (ids mode) responses.

    • ---

    Consumer Usage

    Production base URL: https://pipeagent.dev. Paths below use /api/v1/feed/{feed_id}.

    1. Basic Fetch

    Returns the Singleton object or the latest page of a Collection/Stream.

    bash
    GET https://api.pipeagent.dev/api/v1/feed/{feed_id}

    1.1 Stream Time Window (Native Range)

    For stream feeds, you can query by event time window using inclusive bounds on event_timestamp.

    bash
    GET https://api.pipeagent.dev/api/v1/feed/{feed_id}?start_time=2026-03-24T00:00:00Z&end_time=2026-03-24T23:59:59Z
  • start_time: inclusive lower bound (event_timestamp >= start_time)
  • end_time: inclusive upper bound (event_timestamp <= end_time)
  • Both parameters must be valid ISO-8601 datetime strings.
  • You can combine them with cursor and limit.

    • 2. Advanced JSONPath Projection

    Slice and project JSON on the server to protect your Agent's context window.

    bash
    # Array Slicing (Get top 5)
GET https://api.pipeagent.dev/api/v1/feed/{feed_id}?jsonpath=$[0:5]

# Deep Field Extraction
GET https://api.pipeagent.dev/api/v1/feed/{feed_id}?jsonpath=$[*].metadata.tags

    3. Details Mode

    Retrieve full payloads for specific record IDs identified during browsing.

    bash
    GET https://api.pipeagent.dev/api/v1/feed/{feed_id}?ids=id1,id2

    Performance Tips

  • Use Collection for any list over 20 items.
  • Use Stream for data that grows over time.
  • Always use JSONPath to shield your LLM contexts from irrelevant fields.
    • Version 1.0.4 - Premium Infrastructure