Docs/consumer / integration crewai

CrewAI Integration Guide

CrewAI allows you to coordinate multiple AI agents. This guide shows how to equip your "Researcher" or "Analyst" agents with high-quality PipeAgent data.

1. Prerequisites

Python 3.9+

pip install crewai langchain-openai requests

2. Creating a Custom CrewAI Tool

CrewAI tools are built on top of LangChain tools. Here is the recommended pattern:

python

import os
import requests
from crewai import Agent, Task, Crew, Process
from langchain.tools import tool

# 1. Define the PipeAgent Tool
class PipeAgentTools():
    @tool("market_data")
    def fetch_market_data(jsonpath: str = "$.data"):
        """
        Fetches the latest market rankings and technical data from PipeAgent.
        Ideal for market research and competitive analysis tasks.
        """
        url = "https://api.pipeagent.dev/api/v1/feed/YOUR_FEED_ID"
        headers = {"Authorization": f"Bearer {os.environ['PIPEAGENT_API_KEY']}"}
        
        response = requests.get(url, headers=headers, params={"jsonpath": jsonpath})
        return response.text

# 2. Assign the Tool to an Agent
researcher = Agent(
  role='Senior Market Researcher',
  goal='Discover emerging trends in the AI industry',
  backstory='Expert in analyzing structured data feeds to find hidden market gems.',
  tools=[PipeAgentTools.fetch_market_data],
  verbose=True
)

# 3. Create a Task
task1 = Task(description='Analyze the latest AI tool rankings and identify top 3 competitors.', agent=researcher)

# 4. Form the Crew
crew = Crew(
  agents=[researcher],
  tasks=[task1],
  process=Process.sequential
)

result = crew.start()
print(result)

3. Best Practices for CrewAI

Role Specialization: Only give PipeAgent tools to agents that need them (e.g., a "Researcher" needs data, but a "Writer" might just need the "Researcher's" summary).

JSONPath for Summaries: Use JSONPath to return only the "Summary" or "Title" fields if your crew is doing a broad sweep, to keep the context window clear for other agents.

Dynamic Feed IDs: You can create a tool that first searches via https://api.pipeagent.dev/api/v1/search?q=... and then fetches the best-matching feed with https://api.pipeagent.dev/api/v1/feed/{id}.

4. Comparison

Feature	PipeAgent + CrewAI	Traditional Scrapping + CrewAI
Setup Time	< 5 Minutes	Hours of selector debugging
Reliability	High (Managed Feeds)	Low (Site changes break selectors)
Cost	Fixed per-request	Variable (Proxy & Compute costs)
Agent Performance	High (Structured Data)	Medium (Messy HTML reduces accuracy)

Version 1.0.4 - Premium Infrastructure

Legal Disclaimer

CrewAI Integration Guide

1. Prerequisites

2. Creating a Custom CrewAI Tool

3. Best Practices for CrewAI

4. Comparison