Crewai 课程一些杂记

The 80/20 Rule: Focus on Tasks Over Agents
When building effective AI systems, remember this crucial principle: 80% of your effort
should go into designing tasks.
Even the most perfectly defined agent will fail with poorly designed tasks, but
well-designed tasks can elevate even a simple agent.

● 80% effort:
○ Craft clear tasks

● 20% effort:
○ Polish agent personas

Pitfalls(陷阱)

1. Not spending time on planning use cases
2. Not clear definition of success
3. Not breaking the process into smaller chunks
4. Not measuring / evaluating

Tactics for Debugging, Observing, Optimizing

• TESTING
• Training
• Guardrails

Why Agent Design Matters

• Output quality
  Well-designed agents produce more relevant, high-quality results
• Collaboration effectiveness
  Agents with complementary skills work together more efficiently
• Task performance
  Agents with clear roles and goals execute tasks more effectively
• System scalability
  Thoughtfully designed agents can be reused across multiple crews and contexts

How to provide deterministic controls on probabilistic systems?

• Memory
  Dynamically update context to help agents learn and get better over time

  ---

  How to provide agents the ability to remember information?
  – Short Term
  Stores data from past executions to add context that gets shared among agents
  – Long Term
  Reflects on differences between expect outputs and actual outputs on tasks to improve agents through feedback
  – Entity
  Collects facts about recognizable people, companies, locations, products, etc

  ---

  Agentic Memory
  ● Internal information adapted from previous executions
  ● Selectively added to agent’s context during current execution
  ● Updated from feedback by human users or LLM-as-a-Judge
  Agentic Knowledge
  ● External information retrieved from different sources
  ● Selectively added to agent’s context from flat files or vector databases
  ● Not updated from feedback. Pre-filled at run-time.

  ---

• Guardrails
  Adding either deterministic or probabilistic (LLM as judge) checks on output
  ● Probabilistic Guardrails / LLM Guardrail: LLM as a judge
  ● Deterministic Guardrails / Code Guardrail: Traditional Code

• Hooks
  Execute deterministic code either before or after Agents
  before:
  ● Fetch input data
  ● Clean input data
  ● Check inputs for PII
  …
  after:
  ● Validate outputs
  ● Moderate output content
  ● Log outputs
  …

How to configure tools for reliable run-time behavior?
– Force Return
Directly return the output of tool by specifying return_direct=True in agent
– Rate Limits
Use retry logic and a max usage limit to help agents recover from temporary failures while preventing infinite loops
– Tool Repository
Promotes reuse and sharing of tools across multiple agents and tasks

Before using an MCP server, you must trust it!
● SSE transports can be vulnerable if not properly secured.
● Always validate Origin headers on incoming SSEconnections
● Avoid binding servers to all interfaces locally - bind only to localhost instead
● Implement proper authentication for all SSE connections

Without these protections, attackers could use DNS rebinding to interact with local MCP servers from remote websites.

Mental Models for Agentic Systems

• Agents : Real-time planning
• Graphs : Nodes and edges
• Events : Trigger-based workflows

State provides shared context across each step of your flow

• During Execution
  ● Each step of the flow consists of a function with access to state
  ● All functions can read / write state throughout execution
  ● Accumulated data in state can inform routing of the flow

• After Execution
  ● Optionally you can persist state. Persistence store state for later use.
  ● Note persistent state of flows is different from the memory of crews in the flow
  ● Using persistence is especially important with conversational agents

How to Build Agents you Trust
Flows
Guardrails
Reasoning agents
Human-in-the-loop oversight
Testing
Training
Structured output
Safe code execution

Reliable agents aren’t just accurate,they’re predictable, measurable, and recoverable
Observability: Debuggability , Quality Monitoring
Security: Data Governance, Prompt Injection Safeguards,Secure Code Generation
Compliance: Protecting PII

Common Success Patterns
● High Frequency
● Consensus on Evaluation Criteria
● Reasonable Fallback Paths
● Measurable Outcomes

Common Failure Modes
● Ill-defined goals
● Lack of observability
● No clarity on evaluation criteria
● No way of tracking proper ROI
● No owner for iteration or QA

Open Source
● can be fine-tuned for specific application domains
● can be run locally for privacy protected workflows
● can reduce usage costs and avoid rate limits

Closed Source
● offer advanced reasoning models for complex tasks
● provide safety and robustness features for predictable behavior
● allow for easy integration through managed services

Don’t chase automation Build reliability
● Ease of Use
● Repeatable Outcomes
● Scalable Solutions