Docs/consumer / limits

Usage Limits & Best Practices

To ensure the stability and performance of the pipeAgent gateway, we implement tiered rate limiting and usage policies.

1. Rate Limiting

pipeAgent uses a tiered rate-limiting strategy based on the type of feed you are consuming. All limits are enforced on a per-user, per-feed basis.

Tier	Rate Limit (RPM)	Burst
Free Feeds	60 RPM	10
Paid Feeds	600 RPM	50

Rate Limit Headers

Every response from the pipeAgent API includes headers to help you manage your throughput:

X-RateLimit-Limit: The maximum number of requests allowed in the window.

X-RateLimit-Remaining: The number of requests remaining in the current window.

X-RateLimit-Reset: The time at which the current rate limit window resets (UTC Epoch).

Handling 429 Errors

If you exceed your limit, the API will return a 429 Too Many Requests status code.

💡

TIP

Best Practice: Implement Exponential Backoff in your agent's retry logic. Use the retry_after_ms field in the JSON response to calculate your wait time.

2. Pricing & Credits

pipeAgent uses a Micro-Credit system for paid feeds.

Standard Pricing: Most paid feeds cost $0.001 per 1,000 calls.

Credit Value: 1 Credit = $0.000001 USD.

Promotion Credits: Users with a valid promotion code may receive free credits to test paid feeds.

If your balance is insufficient for a paid feed, the API will return a 402 Payment Required error with a link to the billing dashboard.

3. Query Optimization

To save on latency and data processing costs, use the following optional parameters:

Pagination

If a feed returns an array, use limit and offset to keep payloads small.

limit: Maximum 100 items per request (default: 20 on some endpoints; values above 100 are capped at 100).

offset: Defaults to 0.

cursor: Opaque pagination cursor for collection and stream feeds.

Stream Time Range

For stream feeds, use native time-window filters on event_timestamp:

start_time: inclusive lower bound (event_timestamp >= start_time)

end_time: inclusive upper bound (event_timestamp <= end_time)

Both values must be valid ISO-8601 datetime strings.

bash

GET https://api.pipeagent.dev/api/v1/feed/{id}?limit=20&start_time=2026-03-24T00:00:00Z&end_time=2026-03-24T23:59:59Z

JSONPath Filtering

You can filter the JSON response at the edge using the jsonpath parameter. This prevents your AI agent from receiving irrelevant "noise" and saves LLM token costs.

bash

GET https://api.pipeagent.dev/api/v1/feed/{id}?jsonpath=$[*].{title: @.name, cost: @.price}

4. Provider push (collection, stream & singleton)

When using POST https://api.pipeagent.dev/api/v1/provider/feed/{id}/push, the gateway enforces raw HTTP body size (bytes on the wire) plus, for collection and stream feeds, per-row and row-count guardrails. Limits below are defaults; deployments may override them with environment variables.

Singleton feeds

Guardrail	Default	Environment variable
Max request body	4 MB	`PROVIDER_PUSH_SINGLETON_MAX_BODY_BYTES`

If the body is too large, use a collection feed, split the data, or trim fields.

Collection & stream feeds

Guardrail	Default	Environment variable
Max request body	2 MB	`PROVIDER_PUSH_MAX_REQUEST_BODY_BYTES`
Max JSON size per row	50 KB (UTF-8 bytes of `JSON.stringify(row)`)	`PROVIDER_PUSH_MAX_ROW_BYTES`
Max rows per request (abuse / overload protection)	500	`PROVIDER_PUSH_MAX_ROWS`

Normal integrations should not hit the row cap; split large backfills into multiple requests. The per-row and request body limits apply together: many small rows can still hit the body limit before the row cap.

Error shape (413)

Push responses may include a limit field: singleton_body, request_body, row_count, or row_payload, plus numeric hints such as max_bytes, max_rows, or row_index.

5. Error Codes Reference

Code	Meaning	Action
401	Unauthorized	Missing or malformed `x-api-key` header.
402	Payment Required	Paid feed requested, but balance is $0. Link a payment method.
403	Forbidden	API key is valid but does not have `read` permissions for this feed.
404	Not Found	The feed ID does not exist or has not been synced yet.
429	Too Many Requests	Rate limit exceeded. Wait for the window to reset.
500	Internal Error	Something went wrong on our end. Contact support if persistent.
413	Payload Too Large	Push body or a single row exceeds configured limits (see Provider push above).

6. Data Provenance

All pipeAgent responses include an _origin object. This contains a usage_id and a timestamp.

Transparency: You can use the usage_id to audit your billing logs in the dashboard.

License: All data is provided under the PipeAgent Dynamic Data License. Please respect attribution and redistribution constraints. See Data License.

---

*Need higher limits for an enterprise use case? Contact our sales team for a custom quota.*

Version 1.0.4 - Premium Infrastructure

Legal Disclaimer