Apache SkyWalking – GenAI

Blog: Monitoring LLM Applications with SkyWalking 10.4: Insights into Performance and Cost

Sun, 05 Apr 2026 00:00:00 +0000

With the deep penetration of Generative AI (GenAI) into enterprise workflows, developers face a challenging paradox: while powerful LLM capabilities are easily integrated via Spring AI or OpenAI SDKs, the actual performance and reliability of these calls remain largely invisible.

1. The “Black Box” of Cost and Performance: Is the Expensive Model Worth It?

Facing high LLM bills, organizations often only see a total sum paid to a provider, but cannot calculate the “ROI” within the application.

Blind Upgrades: You might switch to a premium flagship model for a better experience. But in your specific business scenario, does paying several times more per token actually yield lower latency or a faster TTFT (Time to First Token)?
Lack of Real-World Benchmarks: Official benchmarks mean little without your real-world business requests. You need to know which model achieves the perfect balance between “Token/Cost Consumption” and “Response Speed” under your actual prompt lengths and concurrency levels.

2. The Vanishing “Golden Timeout”

Many teams set timeouts for LLM calls arbitrarily (e.g., 30s or 60s).

Too Short: During peak periods or long-text generation, requests are frequently interrupted, causing business failure rates to soar.
Too Long: If a provider hangs, requests pile up in memory, blocking execution threads and potentially leading to the collapse of the entire Java application or microservice cluster. Only by mastering the P99/P95 Latency can you set rational timeout policies based on data rather than intuition.

3. The Overlooked Experience Killer: TTFT

In GenAI scenarios, a user’s perception of speed depends less on the total duration of the conversation and more on “when the first word appears.” * A streaming response with a 10s total duration but a 500ms TTFT feels instantaneous.

A non-streaming response with a 5s total duration but a 4s TTFT feels “frozen.” If your observability system only tracks total latency, you miss the core UX metric that explains why users complain about “AI slowness.”

SkyWalking 10.4: A “Digital Dashboard”
From the Application Perspective The Virtual GenAI capability introduced in Apache SkyWalking 10.4 fills this “observability vacuum.” It avoids reliance on external gateways by using application-side probes (like the Java Agent) to collect the most authentic data from the client’s perspective.

Precise Latency Distribution: Multi-dimensional metrics (P50, P90, P99) help visualize LLM fluctuations to inform dynamic timeout strategies.
Core UX Metric — TTFT Monitoring: Native support for first-token latency in streaming calls.
Multi-dimensional Model Profiling: Aligns token usage, estimated cost, and performance across Providers and Models, helping you choose the most cost-effective solution for your specific needs.

Virtual GenAI Observability

Virtual GenAI represents Generative AI service nodes detected by probe plugins. All performance metrics are based on the GenAI Client Perspective.

For instance, the Spring AI plugin in the Java Agent detects the response latency of a Chat Completion request. SkyWalking then visualizes these in the dashboard:

Traffic & Success Rate (CPM & SLA)
Latency & TTFT
Token Usage (Input/Output)
Estimated Cost

Screenshots:

How It Works

When the SkyWalking Java Agent or OTLP probes intercept calls to mainstream AI frameworks (e.g., Spring AI, OpenAI SDK), they report Trace data to the SkyWalking OAP. The OAP aggregates and computes this data to generate performance metrics for both Providers and Models, which are then rendered in the built-in Virtual-GenAI dashboards.

Installation & Configuration

Requirements

SkyWalking Java Agent: >= 9.7
SkyWalking OAP: >= 10.4

Semantic Conventions & Compatibility

SkyWalking Virtual GenAI follows OpenTelemetry GenAI Semantic Conventions. OAP identifies GenAI-related Spans based on:

SkyWalking Java Agent

Spans must be of type Exit, have the SpanLayer attribute set to GENAI, and contain the gen_ai.response.model tag.

OTLP / Zipkin Probes

Spans must contain the gen_ai.response.model tag.

For details, refer to the E2E configurations:

GenAI Estimated Cost Configuration

Overview

SkyWalking provides a built-in GenAI Billing Configuration File.

This file defines how SkyWalking maps model names from Trace data to their corresponding providers and estimates the token cost for each LLM call. The estimated cost is displayed in the SkyWalking UI alongside trace and metric data, helping users intuitively understand the financial impact of their GenAI usage.

Important: The pricing in this file is intended for cost estimation only and must not be treated as actual billing or invoice amounts. Users are advised to regularly verify the latest rates on the providers’ official pricing pages.

Configuration Structure

Top-level Fields

Field	Type	Description
`last-updated`	`date`	The last update date of the pricing data. All prices are based on public billing standards announced by providers prior to this date.
`providers`	`list`	List of GenAI provider definitions. Each entry contains matching rules and specific model pricing information.

Provider Definition

Each entry under providers defines a GenAI provider:

providers:
- provider: <provider-name>
  prefix-match:
    - <prefix-1>
    - <prefix-2>
  models:
    - name: <model-name>
      aliases: [<alias-1>, <alias-2>]
      input-estimated-cost-per-m: <cost>
      output-estimated-cost-per-m: <cost>

Field	Type	Required	Description
`provider`	`string`	Yes	The provider identifier (e.g., `openai`, `anthropic`, `gemini`). It is displayed as the Virtual GenAI service name in SkyWalking.
`prefix-match`	`list[string]`	Yes	A list of prefixes used to match model names to this provider. If a model name in the Trace data starts with any of these prefixes, it will be mapped to this provider.
`models`	`list[model]`	No	A list of model definitions containing pricing information. If omitted, the system can still identify the provider but will not perform cost estimation.

Model Definition

Each entry under models defines the pricing for a specific model:

Field	Type	Required	Description
`name`	`string`	Yes	The standard model name used for matching.
`aliases`	`list[string]`	No	Alternative names that should resolve to the same billing entry. This is useful when providers use different naming conventions (see the “Model Aliases” section).
`input-estimated-cost-per-m`	`float`	No	Estimated cost per 1,000,000 (one million) input (Prompt) tokens. The default unit is USD.
`output-estimated-cost-per-m`	`float`	No	Estimated cost per 1,000,000 (one million) output (Completion) tokens. The default unit is USD.

Model Matching Mechanism

Provider-Level Prefix Matching

When SkyWalking receives a Trace containing a GenAI call, it determines the Provider based on the following priority order:

gen_ai.provider.name tag: This tag is retrieved first. It follows the latest OpenTelemetry GenAI semantic conventions.
gen_ai.system tag: If the above tag is missing, the system falls back to this legacy tag. Note: This tag is only parsed when processing OTLP or Zipkin format data, primarily for compatibility with older versions of libraries like the Python auto-instrumentation.
Prefix Matching: If neither of the above tags exists, SkyWalking reads the prefix-match rules defined in gen-ai-config.yml and attempts to identify the provider by matching the Model Name.

- provider: openai
  prefix-match:
    - gpt

Any model name starting with gpt (such as gpt-4o, gpt-4.1-mini, or gpt-5-nano) will be mapped to the openai provider. A single provider can have multiple prefixes:

- provider: tencent
  prefix-match:
    - hunyuan
    - Tencent

Model-level Longest-Prefix Matching

Once the provider is determined, SkyWalking uses a Trie-based longest-prefix matching algorithm to find the best billing entry. This is crucial because model names returned in provider API responses often include version numbers or timestamps, differing from the base model name in the config. Example OpenAI config:

models:
- name: gpt-4o
  input-estimated-cost-per-m: 2.5
  output-estimated-cost-per-m: 10.0
- name: gpt-4o-mini
  input-estimated-cost-per-m: 0.15
  output-estimated-cost-per-m: 0.6

Matching behavior:

Model Name in Trace	Matched Configuration Entry	Reason
`gpt-4o`	`gpt-4o`	Exact match
`gpt-4o-2024-08-06`	`gpt-4o`	Longest prefix is `gpt-4o`
`gpt-4o-mini`	`gpt-4o-mini`	Exact match (Longer prefix `gpt-4o-mini` takes priority over `gpt-4o`)
`gpt-4o-mini-2024-07-18`	`gpt-4o-mini`	Longest prefix is `gpt-4o-mini`

This mechanism ensures versioned API model names map to the correct pricing tier without requiring exact full names in the configuration file.

Model Aliases

Some providers use different naming conventions across API responses and documentation. For example, Anthropic’s model might appear as claude-4-sonnet or claude-sonnet-4. The aliases field supports both formats under a single billing entry:

- name: claude-4-sonnet
  aliases: [claude-sonnet-4]
  input-estimated-cost-per-m: 3.0
  output-estimated-cost-per-m: 15.0

Under this configuration, claude-4-sonnet and claude-sonnet-4 (as well as any versioned variants, such as claude-sonnet-4-20250514) will resolve to the same billing entry.
Note: Aliases also participate in longest prefix matching. Therefore, claude-sonnet-4-20250514 will match the alias claude-sonnet-4, which in turn resolves to the pricing information for claude-4-sonnet.

Custom Configuration

Adding a New Provider

To add a provider that is not included in the default configuration:

providers:
# ... Existing providers ...

- provider: ollama
  prefix-match:
    - mymodel
  models:
    - name: mymodel-large
      input-estimated-cost-per-m: 1.0
      output-estimated-cost-per-m: 5.0
    - name: mymodel-small
      input-estimated-cost-per-m: 0.1
      output-estimated-cost-per-m: 0.5

For OTLP/Zipkin data, a dedicated estimated tag has been added. You can now view the cost of each GenAI call directly on the UI.

Main Metrics

1.Provider Level

Metric ID	Description	Meaning
`gen_ai_provider_cpm`	Calls Per Minute	Requests per minute (Throughput)
`gen_ai_provider_sla`	Success Rate	Request success rate
`gen_ai_provider_resp_time`	Avg Response Time	Average response time
`gen_ai_provider_latency_percentile`	Latency Percentiles	Response time percentiles (P50, P75, P90, P95, P99)
`gen_ai_provider_input_tokens_sum/avg`	Input Token Usage	Total and average input token usage
`gen_ai_provider_output_tokens_sum/avg`	Output Token Usage	Total and average output token usage
`gen_ai_provider_total_estimated_cost/avg`	Estimated Cost	Total estimated cost and average cost per call

2. Model Level

Metric ID	Description	Meaning
`gen_ai_model_call_cpm`	Calls Per Minute	Requests per minute for this specific model
`gen_ai_model_sla`	Success Rate	Model-specific request success rate
`gen_ai_model_latency_avg/percentile`	Latency	Average and percentiles of model response duration
`gen_ai_model_ttft_avg/percentile`	TTFT	Time to First Token (Streaming only)
`gen_ai_model_input_tokens_sum/avg`	Input Token Usage	Detailed input token consumption for the model
`gen_ai_model_output_tokens_sum/avg`	Output Token Usage	Detailed output token consumption for the model
`gen_ai_model_total_estimated_cost/avg`	Estimated Cost	Estimated total cost and average cost for the model

Recommended Usage Scenarios

Performance Evaluation: Use Latency and Time to First Token (TTFT) metrics to analyze model inference efficiency and the end-user interaction experience.
Token Monitoring: Real-time monitoring of Input and Output token consumption to analyze resource utilization across different business scenarios.
Cost Alerting: Set alert thresholds based on Estimated Cost or token consumption to promptly detect abnormal calls and prevent budget overruns.

Zh: 基于 SkyWalking 10.4 的大模型应用监控：洞察 LLM 的性能与成本

Sun, 05 Apr 2026 00:00:00 +0000

问题：当应用开始“吞噬”大模型，监控却留下了盲区

随着生成式 AI（GenAI）在企业业务中的深度渗透，开发者正面临一个尴尬的局面：我们在应用中通过Spring AI或OpenAI SDK快速集成了强大的大模型能力，但对于这些调用的实际表现却几乎一无所知。

成本与性能的“黑盒”：昂贵的模型真的更具性价比吗？
面对高昂的大模型账单，我们往往只知道把钱交给了某个Provider，却算不清这笔账在应用内部的“投入产出比”。盲目的选型升级：为了追求更好的体验，你可能将业务默认切换到了成本更高的旗舰模型。但在具体的业务场景下，花费数倍的 Token 成本，它真的能在真实请求中带来更低的延迟和更快的 TTFT(Time to First Token) 吗？缺乏真实的评估基准：脱离了真实的业务请求，单纯看官网的 Benchmark 意义不大，你需要知道在实际的 Prompt 长度和并发压力下，同一Provider下的哪个模型能在“Token/Cost 消耗”与“响应速度”之间达到完美的平衡。如果没有应用侧的数据支撑，你根本无从判断哪款模型才是当前业务的最优解。
消失的“黄金超时时间”
很多团队在代码里给 LLM 调用设置超时（Timeout）时，往往是拍脑袋决定（比如 30s 或 60s）。
设太短：长文本生成或模型高峰期时，请求会被频繁强行中断，导致业务失败率飙升。
设太长：如果下游供应商出现故障（卡死），大量的请求会堆积在应用内存中，阻塞执行线程，最终导致整个 Java 应用甚至微服务集群的瘫痪。只有真正掌握了预估的整体调用延迟（P99/P95 Latency），你才能基于数据而非直觉，为不同模型设置最合理的超时策略。
被忽视的体验杀手：TTFT
在 GenAI 场景下，用户对“快”的感知并不完全取决于整个对话结束的总耗时，而取决于**“第一行字什么时候跳出来”**。一个总耗时 10 秒但 TTFT 仅 500ms 的流式响应，给用户的观感是“秒回”。一个总耗时 5 秒但 TTFT 需要 4s 的非流式响应，给用户的观感却是“卡死”。如果你的观测系统只能看到总耗时，你就会漏掉最核心的 UX 指标，无法解释为什么用户反馈“AI 很慢”即便总耗时看起来还行。

SkyWalking 10.4：应用视角的“数字仪表盘”
Apache SkyWalking 自 10.4 版本引入的 Virtual GenAI 能力，正是为了解决应用层侧的这种“观测真空”。它不依赖任何外部网关，直接通过应用侧探针（如 Java Agent）在客户端视角采集最真实的数据。

精准的延迟分布（Latency Percentiles）：通过 P50、P90、P99 等多维指标，帮你勾勒出 LLM 调用的真实波动曲线，为设置“动态超时时间”提供科学依据。
核心 UX 指标——TTFT 监控：原生支持流式（Streaming）调用的首字延迟统计。通过对比不同 Provider 或不同模型的 TTFT，你可以优化提示词（Prompt）策略或切换更快的模型，确保用户体验始终在线。
多维度的模型“画像”分析：在 Provider 和 Model 两个维度上，将 Token 消耗、预估成本与性能指标深度对齐。这让你不再看供应商全网的“理想平均数”，而是看清你的应用在调用特定模型时的真实表现，从而在复杂的模型生态中选出最具性价比的选型方案。

虚拟 GenAI 观测

虚拟 GenAI 代表了由探针插件检测到的生成式 AI 服务节点。GenAI 操作的性能指标均基于 GenAI 客户端视角。

例如，Java 探针中的 Spring AI 插件可以检测一次对话补全（Chat Completion）请求的响应延迟。随后，SkyWalking 将在仪表盘中展示：

流量与成功率 (CPM & SLA)
响应延迟 (Latency & TTFT)
Token 消耗 (Input/Output)
预估成本 (Estimated Cost)

如图：

原理

当 SkyWalking Java Agent 或 OTLP 探针拦截到主流 AI 框架（如 Spring AI、OpenAI SDK 等）的调用时，将Trace 数据上报至 SkyWalking OAP。 OAP会基于这些 Trace 自动完成数据的聚合与计算。最终会生成 Provider（服务商）与 Model（模型）两个维度的各类性能指标，并直接渲染填充至内置的 Virtual-GenAI 仪表盘中。

安装配置

要求

版本要求

● SkyWalking Java Agent: >= 9.7 ● SkyWalking Oap: >= 10.4

语义规范与兼容性

SkyWalking 虚拟 GenAI 遵循 OpenTelemetry GenAI 语义规范。OAP 将根据以下标准识别 GenAI 相关 Span：

SkyWalking Java Agent

上报的 Span 必须为 Exit 类型，其 SpanLayer 属性需设定为 GENAI,包含gen_ai.response.model 标签。

输出OTLP / Zipkin格式数据的探针

上报的 Span 中包含 gen_ai.response.model 标签。

具体可以参考e2e配置
SkyWalking Java Agent上报数据
 探针上报OTLP格式数据
 探针上报Zipkin格式数据

GenAI 预估成本配置

概览

SkyWalking 提供了一个内置的GenAI计费配置文件

该配置定义了SkyWalking 如何将 Trace 数据中的模型名称映射到对应的供应商，并估算每次 LLM 调用的 Token 成本。估算成本将与 Trace 和指标数据一起显示在 SkyWalking UI 中，帮助用户直观了解 GenAI 使用带来的预估费用影响。重要提示: 此文件中的定价仅用于成本估算，不得视为实际账单或发票金额。建议用户定期从供应商官方定价页面核实最新费率。

配置结构

Top 字段

字段	类型	描述
`last-updated`	`date`	定价数据的最后更新日期。所有价格均基于该日期前各厂商官网公布的公开计费标准。
`providers`	`list`	GenAI 厂商定义列表。每个厂商条目下包含匹配规则（matching rules）以及具体的模型计费信息（model pricing）。

provider 定义

providers 下的每个条目定义一个 GenAI 供应商：

providers:
- provider: <provider-name>
  prefix-match:
    - <prefix-1>
    - <prefix-2>
  models:
    - name: <model-name>
      aliases: [<alias-1>, <alias-2>]
      input-estimated-cost-per-m: <cost>
      output-estimated-cost-per-m: <cost>

字段 (Field)	类型 (Type)	必填 (Required)	描述 (Description)
`provider`	`string`	是	供应商标识（如 `openai`, `anthropic`, `gemini`）。在 SkyWalking 中作为虚拟 GenAI 服务名显示。
`prefix-match`	`list[string]`	是	用于将模型名称匹配到该供应商的前缀列表。如果 Trace 数据中的模型名以其中任一前缀开头，则会被映射到该供应商。
`models`	`list[model]`	否	包含定价信息的模型定义列表。如果省略，系统仍能识别供应商，但不会进行成本估算。

model 定义

models 下的每个条目定义特定模型的定价：

字段 (Field)	类型 (Type)	必填 (Required)	描述 (Description)
`name`	`string`	是	用于匹配的标准模型名称。
`aliases`	`list[string]`	否	应解析为同一计费条目的备选名称。当供应商使用不同的命名习惯时非常有用（参见“模型别名”部分）。
`input-estimated-cost-per-m`	`float`	否	每 1,000,000（一百万）输入（Prompt）Token 的预估成本。默认单位为 USD。
`output-estimated-cost-per-m`	`float`	否	每 1,000,000（一百万）输出（Completion）Token 的预估成本。默认单位为 USD。

模型匹配机制

供应商级前缀匹配

当 SkyWalking 接收到包含 GenAI 调用的 Trace 时，会按照以下优先级顺序来确定供应商（Provider）：

gen_ai.provider.name 标签：首先检索此标签。它是OpenTelemetry最新的语义规范。
gen_ai.system 标签：如果缺少上述标签，系统将回退到此旧版（Legacy）标签。注意：此标签仅在处理 OTLP 或 Zipkin 协议的数据时会被解析，主要用于兼容旧版的 Python 自动仪表化等库。
前缀匹配 (Prefix Matching)：若上述两个标签均不存在，SkyWalking 会读取 gen-ai-config.yml 中定义的 prefix-match 规则，通过匹配模型名称 (Model Name) 来尝试识别供应商。

- provider: openai
  prefix-match:
    - gpt

任何以 gpt 开头的模型名称（如 gpt-4o, gpt-4.1-mini, gpt-5-nano）都会被映射到 openai 供应商。一个供应商可以拥有多个前缀：

- provider: tencent
  prefix-match:
    - hunyuan
    - Tencent

模型级最长前缀匹配 (Model-Level Longest-Prefix Matching)

一旦确定了供应商，SkyWalking 会使用基于前缀树 (Trie) 的最长前缀匹配算法来查找最佳的模型计费条目。这至关重要，因为 LLM 供应商在 API 响应中返回的模型名称通常包含版本号或时间戳，与配置中的基础模型名称有所不同。示例：假设 OpenAI 的配置条目如下：

models:
- name: gpt-4o
  input-estimated-cost-per-m: 2.5
  output-estimated-cost-per-m: 10.0
- name: gpt-4o-mini
  input-estimated-cost-per-m: 0.15
  output-estimated-cost-per-m: 0.6

其匹配行为如下表所示：

Trace 中的模型名称	匹配的配置条目	原因
`gpt-4o`	`gpt-4o`	完全匹配
`gpt-4o-2024-08-06`	`gpt-4o`	最长前缀为 `gpt-4o`
`gpt-4o-mini`	`gpt-4o-mini`	完全匹配（比 `gpt-4o` 更长的前缀优先）
`gpt-4o-mini-2024-07-18`	`gpt-4o-mini`	最长前缀为 `gpt-4o-mini`

这种机制确保了 API 返回的带有版本的模型名称能够被正确映射到相应的价格档位，而无需在配置文件中维护精确的全名。

模型别名 (Model Aliases)

部分供应商在 API 响应和官方文档中会使用不同的命名规范。例如，Anthropic 的模型在 Trace 中可能显示为 claude-4-sonnet 或 claude-sonnet-4。通过 aliases 字段，可以让单个计费条目同时支持这两种配置：

- name: claude-4-sonnet
  aliases: [claude-sonnet-4]
  input-estimated-cost-per-m: 3.0
  output-estimated-cost-per-m: 15.0

在这种配置下，claude-4-sonnet 和 claude-sonnet-4（以及任何带有版本的变体，如 claude-sonnet-4-20250514）都会解析为同一个计费条目。
注意：别名同样参与最长前缀匹配。因此，claude-sonnet-4-20250514 会匹配到别名 claude-sonnet-4，进而解析到 claude-4-sonnet 的定价信息。

自定义配置

添加新供应商 (Adding a New Provider) 要添加默认配置中未包含的供应商：

providers:
# ... 现有供应商 ...

- provider: ollama
  prefix-match:
    - mymodel
  models:
    - name: mymodel-large
      input-estimated-cost-per-m: 1.0
      output-estimated-cost-per-m: 5.0
    - name: mymodel-small
      input-estimated-cost-per-m: 0.1
      output-estimated-cost-per-m: 0.5

针对OTLP/zipkin的数据，新增了单独的estimated tag, 可以在UI上看到这次GenAI调用消耗的cost。

主要指标

1. Provider Level (服务商维度)

指标 ID	描述	含义
`gen_ai_provider_cpm`	Calls Per Minute	每分钟请求数 (吞吐量)
`gen_ai_provider_sla`	Success Rate	请求成功率
`gen_ai_provider_resp_time`	Avg Response Time	平均响应耗时
`gen_ai_provider_latency_percentile`	Latency Percentiles	响应耗时百分位数 (P50, P75, P90, P95, P99)
`gen_ai_provider_input_tokens_sum/avg`	Input Token Usage	输入 Token 的总和及平均值
`gen_ai_provider_output_tokens_sum/avg`	Output Token Usage	输出 Token 的总和及平均值
`gen_ai_provider_total_estimated_cost/avg`	Estimated Cost	预估总成本及次均成本

2. Model Level (模型维度)

指标 ID	描述	含义
`gen_ai_model_call_cpm`	Calls Per Minute	该特定模型的每分钟请求数
`gen_ai_model_sla`	Success Rate	模型请求成功率
`gen_ai_model_latency_avg/percentile`	Latency	模型响应耗时的平均值及百分位数
`gen_ai_model_ttft_avg/percentile`	TTFT	首个token响应时间 (仅限流式传输 Streaming)
`gen_ai_model_input_tokens_sum/avg`	Input Token Usage	该模型的输入 Token 消耗详情
`gen_ai_model_output_tokens_sum/avg`	Output Token Usage	该模型的输出 Token 消耗详情
`gen_ai_model_total_estimated_cost/avg`	Estimated Cost	该模型的预估总成本及次均成本

建议使用场景

性能评估：利用响应延迟（Latency）和首字响应时间（TTFT）指标，分析模型推理效率及终端用户交互体验。
Token 监控：实时监控输入（Input）与输出（Output）Token 的消耗，用于分析不同业务场景下的资源占用情况。
成本预警：支持基于预估成本（Cost）或 Token 消耗量配置告警阈值，及时发现异常调用，防止成本超支。

Blog: Monitoring Envoy AI Gateway with Apache SkyWalking

Thu, 02 Apr 2026 00:00:00 +0000

LLM traffic is becoming a first-class citizen in production infrastructure. Teams are calling OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Google Gemini — often multiple providers at once. But most organizations have no unified visibility into this traffic:

Token costs spiral without knowing which teams, models, or providers drive the spend. A single misconfigured prompt template can burn through thousands of dollars before anyone notices.
Provider outages cause cascading failures. When OpenAI has a bad hour, your application goes down with it — and you have no failover visibility to understand what happened or switch providers automatically.
No unified metrics across heterogeneous LLM calls. Latency, Time to First Token (TTFT), Time Per Output Token (TPOT), token usage, error rates — each provider reports these differently, if at all. There is no single dashboard to compare them.

This is the same observability gap that microservices faced a decade ago. The solution then was service meshes and API gateways with built-in telemetry. For AI workloads, the answer is an AI gateway.

Why an AI Gateway

Envoy AI Gateway is an open-source AI gateway built on top of Envoy Proxy and Envoy Gateway. It is not a standalone SaaS product or a Python proxy — it is infrastructure-grade software built on the same Envoy that already handles traffic for a large portion of cloud-native deployments.

Key capabilities:

Multi-provider routing — supports 16+ AI providers (OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Google Gemini, Mistral, Cohere, DeepSeek, and more) behind a unified API.
Token-based rate limiting — rate limit by token consumption, not just request count.
Provider fallback — automatic failover when a provider is down or slow.
Model virtualization — abstract model names so applications are decoupled from specific providers.
Two-tier architecture — a reference architecture with a centralized entry gateway (Tier 1) for auth and global routing, and per-cluster gateways (Tier 2) for inference optimization.
CNCF ecosystem native — runs on Kubernetes, composes with existing Envoy filters, WASM plugins, and standard Kubernetes Gateway API resources.

Because Envoy AI Gateway natively emits GenAI metrics and access logs via OTLP following OpenTelemetry GenAI Semantic Conventions, it plugs directly into any OpenTelemetry-compatible backend.

Starting from SkyWalking 10.4.0, the OAP server natively receives and analyzes Envoy AI Gateway’s OTLP metrics and access logs — no OpenTelemetry Collector needed in between.

Data Flow

The AI Gateway pushes telemetry directly to SkyWalking via OTLP gRPC:

Application sends LLM API requests through the Envoy AI Gateway.
Envoy AI Gateway routes requests to AI providers (or local models like Ollama) and records GenAI metrics (token usage, latency, TTFT, TPOT) and access logs.
The gateway pushes metrics and logs via OTLP gRPC directly to SkyWalking OAP on port 11800.
SkyWalking OAP parses metrics with MAL rules and access logs with LAL rules, then stores everything in BanyanDB.

No OpenTelemetry Collector is needed. SkyWalking OAP’s built-in OTLP receiver handles everything.

Try It Locally

This demo uses Ollama as a local LLM backend so you can try everything without an API key. The Envoy AI Gateway CLI (aigw) provides a standalone mode that runs outside Kubernetes — perfect for local testing.

Prerequisites

Docker and Docker Compose
Ollama installed on your host

Step 1: Start Ollama

Start Ollama on all interfaces so Docker containers can reach it:

OLLAMA_HOST=0.0.0.0 ollama serve

Pull a small model for testing:

ollama pull llama3.2:1b

Step 2: Start the Stack

Create a docker-compose.yaml:

services:
  banyandb:
    image: apache/skywalking-banyandb:0.10.0
    container_name: banyandb
    ports:
      - "17912:17912"
    command: standalone --stream-root-path /tmp/stream-data --measure-root-path /tmp/measure-data
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:17913/api/healthz || exit 1"]
      interval: 5s
      timeout: 3s
      retries: 10

  oap:
    image: apache/skywalking-oap-server:10.4.0
    container_name: oap
    depends_on:
      banyandb:
        condition: service_healthy
    ports:
      - "11800:11800"
      - "12800:12800"
    environment:
      SW_STORAGE: banyandb
      SW_STORAGE_BANYANDB_TARGETS: banyandb:17912
    healthcheck:
      test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/localhost/12800' || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 30
      start_period: 60s

  ui:
    image: apache/skywalking-ui:10.4.0
    container_name: ui
    depends_on:
      oap:
        condition: service_healthy
    ports:
      - "8080:8080"
    environment:
      SW_OAP_ADDRESS: http://oap:12800

  aigw:
    image: envoyproxy/ai-gateway-cli:latest
    container_name: aigw
    depends_on:
      oap:
        condition: service_healthy
    environment:
      - OPENAI_BASE_URL=http://host.docker.internal:11434/v1
      - OPENAI_API_KEY=unused
      - OTEL_SERVICE_NAME=my-ai-gateway
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://oap:11800
      - OTEL_EXPORTER_OTLP_PROTOCOL=grpc
      - OTEL_METRICS_EXPORTER=otlp
      - OTEL_LOGS_EXPORTER=otlp
      - OTEL_METRIC_EXPORT_INTERVAL=5000
      - OTEL_RESOURCE_ATTRIBUTES=job_name=envoy-ai-gateway,service.instance.id=aigw-1,service.layer=ENVOY_AI_GATEWAY
    ports:
      - "1975:1975"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    command: ["run"]

Start everything:

docker compose up -d

Wait for all services to become healthy (BanyanDB starts first, then OAP, then UI and AI Gateway):

docker compose ps

The key OTLP configuration on the aigw service:

Env Var	Value	Purpose
`OTEL_SERVICE_NAME`	`my-ai-gateway`	Service name in SkyWalking
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://oap:11800`	SkyWalking OAP gRPC endpoint
`OTEL_EXPORTER_OTLP_PROTOCOL`	`grpc`	OTLP transport
`OTEL_METRICS_EXPORTER`	`otlp`	Enable metrics push
`OTEL_LOGS_EXPORTER`	`otlp`	Enable access log push

The OTEL_RESOURCE_ATTRIBUTES must include:

job_name=envoy-ai-gateway — routing tag for MAL/LAL rules
service.instance.id=<id> — instance identity
service.layer=ENVOY_AI_GATEWAY — routes logs to AI Gateway LAL rules

The MAL and LAL rules are enabled by default in SkyWalking OAP. No OAP-side configuration is needed.

Step 3: Run the Demo App

Create a simple Python application that sends requests through the AI Gateway (app.py). It mixes normal requests, streaming requests (for TTFT/TPOT metrics), and error requests (non-existent model → HTTP 404, always captured by the LAL sampling policy):

import time, random, requests

GATEWAY = "http://localhost:1975"
HEADERS = {"Authorization": "Bearer unused", "Content-Type": "application/json"}

questions = [
    "What is Apache SkyWalking? Answer in one sentence.",
    "What is Envoy Proxy used for? Answer in one sentence.",
    "What are the benefits of an AI gateway? Answer in two sentences.",
    "Explain observability in three sentences.",
]

def chat(model, question, stream=False):
    resp = requests.post(
        f"{GATEWAY}/v1/chat/completions",
        json={"model": model, "messages": [{"role": "user", "content": question}], "stream": stream},
        headers=HEADERS, timeout=60, stream=stream,
    )
    if stream:
        chunks = []
        for line in resp.iter_lines():
            if line:
                chunks.append(line.decode())
        return resp.status_code, f"[streamed {len(chunks)} chunks]"
    return resp.status_code, resp.json()

while True:
    r = random.random()
    if r < 0.2:
        # Error request: non-existent model triggers 404
        status, body = chat("non-existent-model", "hello")
        print(f"[error] model=non-existent-model status={status}")
    elif r < 0.5:
        # Streaming request — generates TTFT and TPOT metrics
        q = random.choice(questions)
        status, info = chat("llama3.2:1b", q, stream=True)
        print(f"[stream] status={status} {info}")
    else:
        # Normal non-streaming request
        q = random.choice(questions)
        status, body = chat("llama3.2:1b", q)
        answer = body.get("choices", [{}])[0].get("message", {}).get("content", "")[:80]
        tokens = body.get("usage", {})
        print(f"[ok] status={status} tokens={tokens} answer={answer}...")
    time.sleep(random.randint(20, 30))

Run it:

pip install requests
python app.py

The application talks to the AI Gateway on port 1975, which routes to Ollama. Each request generates GenAI metrics (token usage, latency, TTFT, TPOT) and access logs that the gateway pushes to SkyWalking via OTLP.

The error requests (non-existent model → HTTP 404) are always captured by the access log sampling policy, so you will see them in the SkyWalking log view.

Step 4: View in SkyWalking UI

Open http://localhost:8080 and select the GenAI > Envoy AI Gateway menu.

The service list shows my-ai-gateway with CPM, latency, and token rates at a glance:

Click into the service to see the full dashboard — Request CPM, Latency (average + percentiles), Input/Output Token Rates, TTFT, and TPOT:

The Providers tab breaks down metrics by AI provider:

The Models tab shows per-model metrics including TTFT and TPOT (streaming only). Note the unknown model entries — these are the error requests with non-existent models:

The Log tab shows access logs. The sampling policy drops normal successful responses but always captures errors (HTTP 404) and high-token requests:

Cleanup

docker compose down

Deploying on Kubernetes

For production deployments, Envoy AI Gateway runs as a full Kubernetes controller with Envoy Gateway as the control plane. See the Envoy AI Gateway getting started guide for Kubernetes installation.

The OTLP configuration is the same — set the OTEL_* environment variables on the AI Gateway’s external processor to point at SkyWalking OAP’s gRPC port (11800). See the SkyWalking Envoy AI Gateway Monitoring documentation for details.

GenAI Observability Without an AI Gateway

Not every deployment uses an AI gateway. If your applications call LLM providers directly, SkyWalking 10.4.0 also provides GenAI observability through the Virtual GenAI layer.

This works with any SkyWalking-instrumented, OpenTelemetry-instrumented, or Zipkin-instrumented application. When traces carry gen_ai.* tags (following OpenTelemetry GenAI Semantic Conventions), SkyWalking derives per-provider and per-model metrics from the client side: latency, token usage, success rate, and estimated cost.

For Java applications, the SkyWalking Java Agent (9.7+) includes a Spring AI plugin that automatically instruments calls to 13+ providers (OpenAI, Anthropic, AWS Bedrock, Google GenAI, DeepSeek, Mistral, etc.) with the correct gen_ai.* span tags — no code changes needed.

This is a different use case from the Envoy AI Gateway monitoring covered above:

Envoy AI Gateway layer: infrastructure-level observability — what the gateway sees across all traffic. Best for platform teams managing centralized AI routing.
Virtual GenAI layer: application-level observability — what each instrumented app sees for its own LLM calls. Best for teams without a centralized gateway, or for per-application cost tracking.

References

Envoy AI Gateway — project site and documentation
Envoy AI Gateway CLI — standalone mode for local development
SkyWalking Envoy AI Gateway Monitoring — OAP setup doc
SkyWalking Virtual GenAI — client-side GenAI observability
OpenTelemetry GenAI Semantic Conventions — the metric/attribute standard both projects follow

Zh: 用 Apache SkyWalking 监控 Envoy AI Gateway

Thu, 02 Apr 2026 00:00:00 +0000

问题：LLM 流量缺乏统一观测

LLM 流量正在成为生产基础设施中不可忽视的一部分。团队同时在调用 OpenAI、Anthropic、AWS Bedrock、Azure OpenAI、Google Gemini——往往还不止一个提供商。但大多数组织对这些流量缺乏统一的可见性：

Token 费用失控，却不知道哪个团队、哪个模型、哪个提供商在烧钱。一个配置不当的 prompt 模板就可能在无人察觉的情况下烧掉几千美元。
提供商故障引发连锁反应。 OpenAI 出问题的那一小时，你的应用也跟着挂——而你既没有故障切换的可见性，也无法自动切换提供商。
缺乏统一指标。 延迟、首 Token 耗时（TTFT）、每 Token 输出耗时（TPOT）、Token 用量、错误率——每个提供商的报告方式都不一样，有些甚至不提供。没有一个统一的面板能做对比。

这和十年前微服务面临的可观测性困境如出一辙。当时的解法是服务网格和内置遥测的 API 网关。对 AI 工作负载来说，答案就是 AI 网关。

为什么选择 AI 网关

Envoy AI Gateway 是一个开源 AI 网关，构建在 Envoy Proxy 和 Envoy Gateway 之上。底层就是云原生世界里已经广泛部署的 Envoy，天然具备基础设施级的稳定性和性能。

核心能力：

多提供商路由 —— 支持 16+ AI 提供商（OpenAI、Anthropic、AWS Bedrock、Azure OpenAI、Google Gemini、Mistral、Cohere、DeepSeek 等），统一 API 接入。
基于 Token 的限流 —— 按 Token 消耗限流，而不只是按请求数。
提供商故障切换 —— 某个提供商宕机或响应慢时自动切换。
模型虚拟化 —— 抽象模型名称，让应用与具体提供商解耦。
两层架构 —— 参考架构包含一个集中入口网关（Tier 1）负责认证和全局路由，以及每集群网关（Tier 2）负责推理优化。
CNCF 生态原生 —— 运行在 Kubernetes 上，兼容现有的 Envoy Filter、WASM 插件和标准 Kubernetes Gateway API 资源。

Envoy AI Gateway 原生支持通过 OTLP 发送 GenAI 指标和访问日志，遵循 OpenTelemetry GenAI 语义约定，可以直接接入任何兼容 OpenTelemetry 的后端。

从 SkyWalking 10.4.0 开始，OAP 原生接收和分析 Envoy AI Gateway 的 OTLP 指标和访问日志——中间不需要部署 OpenTelemetry Collector。

数据流

AI Gateway 通过 OTLP gRPC 直接将遥测数据推送到 SkyWalking：

应用通过 Envoy AI Gateway 发送 LLM API 请求。
Envoy AI Gateway 将请求路由到 AI 提供商（或 Ollama 这样的本地模型），同时记录 GenAI 指标（Token 用量、延迟、TTFT、TPOT）和访问日志。
网关通过 OTLP gRPC 直接将指标和日志推送到 SkyWalking OAP 的 11800 端口。
SkyWalking OAP 用 MAL 规则解析指标、用 LAL 规则解析访问日志，然后统一存储到 BanyanDB。

不需要 OpenTelemetry Collector。SkyWalking OAP 内置的 OTLP 接收器可以直接处理所有数据。

本地体验

这个 Demo 使用 Ollama 作为本地 LLM 后端，不需要任何 API Key 就能跑起来。Envoy AI Gateway CLI（aigw）提供独立运行模式，不依赖 Kubernetes，非常适合本地测试。

前置条件

Docker 和 Docker Compose
主机上已安装 Ollama

第一步：启动 Ollama

让 Ollama 监听所有网络接口，以便 Docker 容器能访问到：

OLLAMA_HOST=0.0.0.0 ollama serve

拉取一个小模型用于测试：

ollama pull llama3.2:1b

第二步：启动服务栈

创建 docker-compose.yaml：

services:
  banyandb:
    image: apache/skywalking-banyandb:0.10.0
    container_name: banyandb
    ports:
      - "17912:17912"
    command: standalone --stream-root-path /tmp/stream-data --measure-root-path /tmp/measure-data
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:17913/api/healthz || exit 1"]
      interval: 5s
      timeout: 3s
      retries: 10

  oap:
    image: apache/skywalking-oap-server:10.4.0
    container_name: oap
    depends_on:
      banyandb:
        condition: service_healthy
    ports:
      - "11800:11800"
      - "12800:12800"
    environment:
      SW_STORAGE: banyandb
      SW_STORAGE_BANYANDB_TARGETS: banyandb:17912
    healthcheck:
      test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/localhost/12800' || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 30
      start_period: 60s

  ui:
    image: apache/skywalking-ui:10.4.0
    container_name: ui
    depends_on:
      oap:
        condition: service_healthy
    ports:
      - "8080:8080"
    environment:
      SW_OAP_ADDRESS: http://oap:12800

  aigw:
    image: envoyproxy/ai-gateway-cli:latest
    container_name: aigw
    depends_on:
      oap:
        condition: service_healthy
    environment:
      - OPENAI_BASE_URL=http://host.docker.internal:11434/v1
      - OPENAI_API_KEY=unused
      - OTEL_SERVICE_NAME=my-ai-gateway
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://oap:11800
      - OTEL_EXPORTER_OTLP_PROTOCOL=grpc
      - OTEL_METRICS_EXPORTER=otlp
      - OTEL_LOGS_EXPORTER=otlp
      - OTEL_METRIC_EXPORT_INTERVAL=5000
      - OTEL_RESOURCE_ATTRIBUTES=job_name=envoy-ai-gateway,service.instance.id=aigw-1,service.layer=ENVOY_AI_GATEWAY
    ports:
      - "1975:1975"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    command: ["run"]

启动所有服务：

docker compose up -d

等待所有服务变为健康状态（BanyanDB 先启动，然后是 OAP，最后是 UI 和 AI Gateway）：

docker compose ps

aigw 服务的关键 OTLP 配置：

环境变量	值	用途
`OTEL_SERVICE_NAME`	`my-ai-gateway`	SkyWalking 中的服务名
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://oap:11800`	SkyWalking OAP gRPC 端点
`OTEL_EXPORTER_OTLP_PROTOCOL`	`grpc`	OTLP 传输协议
`OTEL_METRICS_EXPORTER`	`otlp`	启用指标推送
`OTEL_LOGS_EXPORTER`	`otlp`	启用访问日志推送

OTEL_RESOURCE_ATTRIBUTES 必须包含：

job_name=envoy-ai-gateway —— MAL/LAL 规则的路由标签
service.instance.id=<id> —— 实例标识
service.layer=ENVOY_AI_GATEWAY —— 将日志路由到 AI Gateway LAL 规则

MAL 和 LAL 规则在 SkyWalking OAP 中默认启用，不需要额外配置。

第三步：运行 Demo 应用

创建一个简单的 Python 应用，通过 AI Gateway 发送请求（app.py）。它混合了普通请求、流式请求（用于产生 TTFT/TPOT 指标）和错误请求（不存在的模型 → HTTP 404，始终会被 LAL 采样策略捕获）：

import time, random, requests

GATEWAY = "http://localhost:1975"
HEADERS = {"Authorization": "Bearer unused", "Content-Type": "application/json"}

questions = [
    "What is Apache SkyWalking? Answer in one sentence.",
    "What is Envoy Proxy used for? Answer in one sentence.",
    "What are the benefits of an AI gateway? Answer in two sentences.",
    "Explain observability in three sentences.",
]

def chat(model, question, stream=False):
    resp = requests.post(
        f"{GATEWAY}/v1/chat/completions",
        json={"model": model, "messages": [{"role": "user", "content": question}], "stream": stream},
        headers=HEADERS, timeout=60, stream=stream,
    )
    if stream:
        chunks = []
        for line in resp.iter_lines():
            if line:
                chunks.append(line.decode())
        return resp.status_code, f"[streamed {len(chunks)} chunks]"
    return resp.status_code, resp.json()

while True:
    r = random.random()
    if r < 0.2:
        # Error request: non-existent model triggers 404
        status, body = chat("non-existent-model", "hello")
        print(f"[error] model=non-existent-model status={status}")
    elif r < 0.5:
        # Streaming request — generates TTFT and TPOT metrics
        q = random.choice(questions)
        status, info = chat("llama3.2:1b", q, stream=True)
        print(f"[stream] status={status} {info}")
    else:
        # Normal non-streaming request
        q = random.choice(questions)
        status, body = chat("llama3.2:1b", q)
        answer = body.get("choices", [{}])[0].get("message", {}).get("content", "")[:80]
        tokens = body.get("usage", {})
        print(f"[ok] status={status} tokens={tokens} answer={answer}...")
    time.sleep(random.randint(20, 30))

运行：

pip install requests
python app.py

应用通过 1975 端口与 AI Gateway 通信，AI Gateway 再路由到 Ollama。每次请求都会产生 GenAI 指标（Token 用量、延迟、TTFT、TPOT）和访问日志，由网关通过 OTLP 推送到 SkyWalking。

错误请求（不存在的模型 → HTTP 404）始终会被访问日志采样策略捕获，所以在 SkyWalking 的日志视图中一定能看到。

第四步：在 SkyWalking UI 中查看

打开 http://localhost:8080，选择 GenAI > Envoy AI Gateway 菜单。

服务列表显示 my-ai-gateway，可以一览 CPM、延迟和 Token 速率：

点击进入服务详情，查看完整仪表盘——请求 CPM、延迟（平均值 + 百分位数）、输入/输出 Token 速率、TTFT 和 TPOT：

Providers 标签页按 AI 提供商维度展示指标：

Models 标签页展示每个模型的指标，包括 TTFT 和 TPOT（仅流式请求）。注意 unknown 模型条目——这些就是使用不存在模型的错误请求：

Log 标签页展示访问日志。采样策略会丢弃正常的成功响应，但始终保留错误（HTTP 404）和高 Token 消耗的请求：

清理

docker compose down

Kubernetes 生产部署

生产环境中，Envoy AI Gateway 作为完整的 Kubernetes 控制器运行，以 Envoy Gateway 作为控制面。详见 Envoy AI Gateway 入门指南。

OTLP 配置方式相同——在 AI Gateway 的 External Processor 上设置 OTEL_* 环境变量，指向 SkyWalking OAP 的 gRPC 端口（11800）。详见 SkyWalking Envoy AI Gateway 监控文档。

不用 AI 网关也能做 GenAI 可观测

并非所有场景都需要 AI 网关。如果你的应用直接调用 LLM 提供商，SkyWalking 10.4.0 也提供了基于 Virtual GenAI 层的 GenAI 可观测方案。

任何接入了 SkyWalking、OpenTelemetry 或 Zipkin 探针的应用都能使用这个功能。只要 Trace 中携带 gen_ai.* 标签（遵循 OpenTelemetry GenAI 语义约定），SkyWalking 就能从客户端视角推导出每提供商、每模型的指标：延迟、Token 用量、成功率和预估费用。

对于 Java 应用，SkyWalking Java Agent（9.7+）内置了 Spring AI 插件，自动为 13+ 提供商（OpenAI、Anthropic、AWS Bedrock、Google GenAI、DeepSeek、Mistral 等）的调用注入正确的 gen_ai.* Span 标签——不需要改代码。

这与上面介绍的 Envoy AI Gateway 监控是不同的使用场景：

Envoy AI Gateway 层：基础设施级可观测——网关视角，覆盖所有流量。适合负责集中 AI 路由的平台团队。
Virtual GenAI 层：应用级可观测——每个应用自己看到的 LLM 调用情况。适合没有集中网关的团队，或者需要按应用维度跟踪费用的场景。

参考资料

Envoy AI Gateway —— 项目官网和文档
Envoy AI Gateway CLI —— 本地开发用的独立运行模式
SkyWalking Envoy AI Gateway 监控 —— OAP 配置文档
SkyWalking Virtual GenAI —— 客户端侧 GenAI 可观测
OpenTelemetry GenAI 语义约定 —— 两个项目共同遵循的指标/属性标准

Apache SkyWalking – GenAI

Blog: Monitoring LLM Applications with SkyWalking 10.4: Insights into Performance and Cost

The Problem: As Applications “Consume” LLMs, Monitoring Leaves a Blind Spot

1. The “Black Box” of Cost and Performance: Is the Expensive Model Worth It?

2. The Vanishing “Golden Timeout”

3. The Overlooked Experience Killer: TTFT

Virtual GenAI Observability

How It Works

Installation & Configuration

Requirements

Semantic Conventions & Compatibility

SkyWalking Java Agent

OTLP / Zipkin Probes

GenAI Estimated Cost Configuration

Overview

Configuration Structure

Top-level Fields

Provider Definition

Model Definition

Model Matching Mechanism

Provider-Level Prefix Matching

Model-level Longest-Prefix Matching

Model Aliases

Custom Configuration

Adding a New Provider

Main Metrics

1.Provider Level

2. Model Level

Recommended Usage Scenarios

Zh: 基于 SkyWalking 10.4 的大模型应用监控：洞察 LLM 的性能与成本

问题：当应用开始“吞噬”大模型，监控却留下了盲区

虚拟 GenAI 观测

原理

安装配置

要求

版本要求

语义规范与兼容性

SkyWalking Java Agent

输出OTLP / Zipkin格式数据的探针

GenAI 预估成本配置

概览

配置结构

Top 字段

provider 定义

model 定义

模型匹配机制

供应商级前缀匹配

模型级最长前缀匹配 (Model-Level Longest-Prefix Matching)

模型别名 (Model Aliases)

自定义配置

主要指标

1. Provider Level (服务商维度)

2. Model Level (模型维度)

建议使用场景

Blog: Monitoring Envoy AI Gateway with Apache SkyWalking

The Problem: Flying Blind with LLM Traffic

Why an AI Gateway

Data Flow

Try It Locally

Prerequisites

Step 1: Start Ollama

Step 2: Start the Stack

Step 3: Run the Demo App

Step 4: View in SkyWalking UI

Cleanup

Deploying on Kubernetes

GenAI Observability Without an AI Gateway

References

Zh: 用 Apache SkyWalking 监控 Envoy AI Gateway

问题：LLM 流量缺乏统一观测

为什么选择 AI 网关

数据流

本地体验

前置条件

第一步：启动 Ollama

第二步：启动服务栈

第三步：运行 Demo 应用

第四步：在 SkyWalking UI 中查看

清理

Kubernetes 生产部署