Agentic AI 一些知识
自动化程度（Degrees of autonomy）
Agentic AI 可以从低到高拥有不同的自主程度：
| Less autonomous（低自主） | Semi-autonomous（半自主） | Highly autonomous（高自主） |
|---|---|---|
| • All steps predetermined 所有步骤预先确定 • All tool use hard coded 所有工具调用均为硬编码 • Autonomy is in text generation 自主性仅体现在文本生成阶段 |
• Agent can make some decisions, choose tools Agent 可做部分决策、选择工具 • All tools predefined 所有工具为预先定义 |
• Agent makes many decisions autonomously Agent 可自主做大量决策 • Can create new tools on the fly 可即时动态创建新工具 |
Agentic 工作流的核心优势（Key benefits of agentic workflows）
- Much better performance — 性能显著更优（Coding benchmark HumanEval 对比中，Agentic Systems 明显优于 Non-agentic）
- Faster than humans because of parallelization — 因可并行执行，处理速度比人类更快
- Modular: can add or update tools, swap out models — 模块化设计：可新增/更新工具，或替换底层模型
适用场景（What tasks is agentic AI suited to?）
按从易到难排列：
| Easier（较简单） | Harder（较复杂） |
|---|---|
| Clear, step-by-step process 清晰、逐步可执行的流程 |
Steps not known ahead of time 步骤无法预先确定 |
| Standard procedures to follow 有标准流程可依循 |
Plan/solve as you go 需要边规划边求解 |
| Text assets only 仅涉及文本资源 |
Multimodal (sound, vision) 多模态（声音、视觉等） |
底层构建模块（What building blocks do you have?）
| Building block（构建模块） | Examples（示例） | Use cases（用例） |
|---|---|---|
| Models（模型） | LLMs Other AI models |
Text generation, tool use, information extraction 文本生成、工具使用、信息抽取 PDF-to-text, text-to-speech, image analysis PDF 转文本、语音合成、图像分析 |
| Tools（工具） | API Information retrieval Code execution |
Web search, get real-time data, send email, check calendar,… 网页搜索、实时数据获取、发邮件、查看日历… Databases, Retrieval Augmented Generation (RAG) 数据库、检索增强生成（RAG） Basic calculator, data analysis 基础计算、数据分析 |
评估方法（Evaluating Agentic AI）
- Can evaluate using code (objective evals), or LLM-as-judge (subjective evals)
可通过代码进行客观评估（objective evals），或使用 LLM-as-judge 进行主观评估（subjective evals） - Two types of evals: End-to-end and component-level
两类评估方式：端到端评估（End-to-end）与组件级别评估（component-level） - Examine traces to perform error analysis
通过检查执行链路（traces）来做错误分析 - Much more on evals and error analysis in Module 4!
更详细的评估与错误分析内容见 Module 4
4 种 Agentic 设计模式（Agentic Design Patterns）
- Reflection：反思 — Agent 对自身输出进行审视和修正
- Tool use：工具使用 — 调用外部工具/函数扩展能力
- Planning：规划 — 将复杂任务拆解为有序步骤并逐步执行
- Multi-agent collaboration：多 Agent 协作 — 多个 Agent 协同工作，提升效果与速度
reflection
针对zero-shot prompting 的 LLM 模型，Agentic AI 可以通过reflection模式来改进自身输出。
实验1：输出-反馈-优化-产出
实验2：输出-反馈(with more 输出信息/对输出结果的校验分析)-优化-产出
Evaluating reflection（评估 Reflection）
Objective evals（客观评估）
- Code-based evals are easier — 基于代码的评估更容易执行
- Build a dataset of ground truth examples — 构建真实标注（ground truth）示例数据集
Subjective evals（主观评估）
- Use LLM as a judge — 使用 LLM 作为评审者
- Rubric-based grading is better — 基于评分规则（rubric）的打分方式效果更好
with external feedback
可以借助外部工具进行校验进一步改善reflection效果。
tool use (function calling)
实验1 简单调用 从一个工具到多个工具
实验2 function calling 当做tool用
planning
避免提前进行工作流硬编码，使整个流程更具有通用性，以及实现更高层次的任务处理，可以通过大模型根据用户问题，自行规划任务，所有操作被整理成一系列步骤，然后让大语言模型来执行这些任务
任务list 的表示形式可以为json, xml， markdown 等 语法格式，方便后续的解析和执行。
为避免因为planning 的多样性造成不同步骤都需要提供步骤使用的特定tool 工具，可以尝试让llm 自行进行代码编写和执行
从一个步骤调一个工具函数，转变成让llm 实现一个函数，函数中通过代码一步一步进行任务步骤实现
multi-agent collaboration
串行协作
交给llm 自行规划调用
多智能体通信模式
Evaluating agnetic
Driving your development process with evals
• Build a system and look at outputs to discover where it is behaving in an unsatisfactory way E.g.incorrect due dates in invoice data extract
• Drive improvement by putting in place a small eval with ~20 examples to help you track progress
• Monitor as you make changes to workflow （e.g.new orompts, new algorithms） and see if the metric improves
Create an eval to measure performance
- Choose 3-5 gold standard discussion points for each topic
- Use LLM-as-a-judge to count how many topics were mentioned
- Get score for each prompt in eval set
Tips for designing end-to-end evals
• Quick and dirty is ok to start！
• As you find places where your evals fail to capture human judgement as to what system is better, use that as an opportunity to improve the metric
• Look for places where performance is worse than humans
Tips for error analysis
• Develop a habit of looking at traces
• Carry out error analysis to figure out what component performed poorly, leading to a poor final output
• Use error analysis output to decide where to focus efforts
Benefits of component-level evaluations
end to end evals is more expensive,so if the eval scope scales to smaller, use component-level evals is more efficient
• Can provide clearer signal for specific errors
• Avoid the noise in end-to-end system
• More efficient for focused team to optimize
• Work on smaller, more targeted problems faster
Improving non-LLM component performance
E.g. web search, text retrieval for RAG, code execution, trained ML
model （for speech recognition, people detection, etc.）
• Tune hyperparameters of component
Web search: Number of results, date range
RAG: Change similarity threshold, chunk size
ML models: Detection threshold
• Replace the component
Try a different web search engine, RAG provider, etc.
Improving LLM component performance
• Improve your prompts
Add more explicit instructions.
Add one or more concrete example to the prompt （few-shot prompting）
• Try a new model
Try multiple LLMs and use evals to pick the best A
小模型对于回答简单事实性问题往往表现更好
大模型前沿的模型在遵循指令方面表现更好
• Split up the step
Decompose the task into smaller steps
• Fine-tune a model
Fine tune on your internal data to improve performance
Developing intuition for model intelligence
建立对模型智能模型的直觉
Play with models often
• Having a personal set of evals might be helpful
• Read other people’s prompts for ideas of how to best use models
use different models in your agentic workflows
• Which models work for which types of tasks？
• aisuite makes it easy to quickly swap out
优化延迟和成本问题
通过分析监控workflow 上每个组件的延迟和成本数据 通常可以得到相应的解决方案
Costing your workflow
• LLM steps （pay per token）
• Any APl-calling tools （pay per API call）
• Compute steps （based on server capacity/cost）
开发总流程
