Apache SkyWalking – Release

Blog: Apache SkyWalking 2025 in Review: Making BanyanDB Ready for Production

Thu, 01 Jan 2026 00:00:00 +0000

2025 was a very focused year for the Apache SkyWalking community: moving BanyanDB from “native storage” to a “production-ready default”, and making SkyWalking APM fully benefit from that foundation.

This post summarizes the key milestones, with an emphasis on BanyanDB.

Storage strategy: saying goodbye to H2

We started 2025 with a clear direction: the H2 storage option is permanently removed. This change reduced long-term maintenance burden and removed a storage choice that was not aligned with production and cloud-native deployments.

BanyanDB: from 0.8.0 foundations to 0.9.0 production features

BanyanDB 0.8.0 delivered the “day-2 operations” foundation that a default storage backend needs. The community put a lot of effort into making queries faster and more predictable (for example index_mode, numeric index types, and multiple query-path optimizations), while also making the system safer under real production pressure. Disk-usage thresholds and a query memory protector were added as guardrails, and the operational toolbox matured with snapshot/backup/restore utilities and improved metadata synchronization.

Just as importantly, 0.8.0 started filling in the missing pieces of a full platform: native property storage and lifecycle-related capabilities that later enabled stronger HA and stage-based deployment patterns.

BanyanDB 0.9.0 was the “production features” milestone. It introduced the Trace data model as a first-class citizen, which unlocked much deeper trace storage and query capabilities. On the reliability and scaling side, the release brought configurable replicas, liaison-side improvements (including load balancing and moving some TopN flow), and broader correctness work such as migrations, version compatibility checks, and access logs.

It also made long-term operations more cloud-friendly with backup/restore support for AWS S3, GCS, and Azure Blob Storage, and added authentication primitives needed in shared environments. In short, 0.9.0 is where BanyanDB clearly moved beyond a “fast storage engine” into a “production platform”.

SkyWalking APM: BanyanDB becomes the default path

With APM 10.2.0, the project made the strategic shift official: H2 was removed permanently, and BanyanDB 0.8.0 became the default path that SkyWalking invests in. A lot of the work here was not flashy, but essential — refining OAP behavior (group settings, index model changes, Progressive TTL, query limits, and more) so running BanyanDB in production felt stable and predictable.

With APM 10.3.0, SkyWalking and BanyanDB moved forward together: BanyanDB 0.9.0’s new trace model was adopted end-to-end, reducing inefficient query round-trips and enabling new query views that significantly lowered page latency. The integration also expanded into lifecycle-aware operations with hot/warm/cold stage configuration (including TTL and query support), and added BanyanDB self-monitoring through OAP and the UI — the kind of end-to-end polish that turns a storage backend into a truly native solution.

If you’d like this review to cover APM 10.4.x as well, please point me to the corresponding release content in this repo (I didn’t find an APM 10.4.0 release announcement in the current checkout).

Packaging and deployment ecosystem (Helm)

BanyanDB’s production readiness is not only server features — it also depends on deployment maturity.

Helm charts:
- SkyWalking Kubernetes Helm Chart 4.8.0 improved BanyanDB deployment defaults by updating the bundled BanyanDB Helm dependency, fixing an init-job volume-mount mismatch, and aligning OAP/UI images with the APM 10.3.0 line.
- BanyanDB Helm 0.4.0 added backup/restore sidecars and a default volume for property storage.
- BanyanDB Helm 0.5.0 introduced stage-aware patterns (hot/warm/cold), improved lifecycle-sidecar scheduling, moved liaison to StatefulSet, refined internal networking, and expanded configuration options.
- BanyanDB Helm 0.5.1 refined liaison configuration and fixed restore-init environment issues.
- BanyanDB Helm 0.5.3 fixed a liaison/data-node port issue.

The rest of the community: agents and tooling kept moving

While storage was the “main storyline”, the community shipped releases across agents, clients, and surrounding components throughout 2025.

Below is a consolidated view of the other releases, grouped by project, with the most important notes.

SkyWalking Java Agent
- 9.4.0: agent self-observability; async-profiler support; broader plugin improvements.
- 9.5.0: virtual thread executor plugin; compatibility and stability fixes; dependency upgrades.
SkyWalking Go
- 0.6.0: richer manual APIs (events/logs/metrics, set span error); goframev2 plugin; bug fixes including Redis cluster mode.
SkyWalking for NodeJS
- 0.8.0: Express 4/5 compatibility, keep-alive HTTP trace fix, and test/dependency maintenance.
SkyWalking Python
- 1.2.0: sampling service, sw_grpc plugin, async/profiling stability fixes, Python 3.13 support, and dropping Python 3.7.
SkyWalking PHP
- 1.0.0: reach 1.0; add PSR-3 log reporting; upgrade toolchain/dependencies.
SkyWalking Rust
- 0.9.0: migrate to Rust edition 2024 and upgrade dependencies.
- 0.10.0: Kafka client configuration refactor, rdkafka upgrade, CI maintenance.
SkyWalking Ruby
- 0.1.0: initialize agent core and e2e tests; add plugins for Sinatra, redis-rb, net-http, memcached, and Elasticsearch.
SkyWalking Client JS
- 1.0.0: add Core Web Vitals and static resource metrics; fix fetch/resource error handling; dependency and e2e/test improvements.
SkyWalking Satellite
- 1.3.0: support native eBPF Access Log protocol and async-profiler protocol; upgrade Go toolchain.
SkyWalking Eyes
- 0.7.0: improve installation/docs, respect gitignore behavior, upgrade Go, and simplify release steps.
- 0.8.0: add Elixir support and stronger dependency-license scanning (notably Ruby via Gemfile.lock), plus stability fixes.

Looking ahead: possible directions in 2026

2025 was about making BanyanDB ready for production. In 2026, the community is exploring the next set of improvements that could make the whole stack simpler to operate, more stable under stress, and easier to integrate into broader observability ecosystems.

Possible areas include:

BanyanDB: remove the etcd dependency: the direction under discussion is to move away from etcd (given ecosystem activity and maintenance concerns) and rely more on DNS-based discovery plus BanyanDB’s native property capabilities.
BanyanDB: stronger stability testing: more systematic testing, including chaos testing, to validate behavior under failures and noisy conditions.
BanyanDB: better observability export: introducing First Occurrence Data Collection (FODC) as a sidecar and proxy server to provide a unified stream of observability data to third-party systems.
SkyWalking APM: broader runtime and query capabilities: cold-stage data query support, a newer Java runtime (Java 25), and consideration of TraceQL protocol (Temper) support.

Closing

Thanks to everyone who contributed to SkyWalking in 2025. Every contribution is high-value — code, documentation, reviews, testing, issue triage, and operational experience — and each of them helped move the project forward.

We also want to say a special thank you to the countless end users across global companies. Many of the most valuable improvements don’t start from a pull request: they start from real-world use cases, performance investigations, production feedback, bug reports, and the patience to help us reproduce and validate fixes.

As another milestone, SkyWalking reached 968 GitHub contributors globally, and we expect the 1000th contributor milestone to arrive soon in 2026. But the community is much larger than the number suggests, and SkyWalking’s progress has always been driven by collaboration between contributors, adopters, and maintainers.

Apache SkyWalking was originally created by Sheng Wu as a personal project in May 2015. It would never have grown into what it is today without the whole community — and it will keep moving forward because of the community.

Blog: BanyanDB 0.6 Release: Enhanced Performance and Efficiency

Tue, 04 Jun 2024 00:00:00 +0000

Introduction

We are excited to announce the release of BanyanDB v0.6, a significant milestone in the evolution of our database technology. This latest version introduces a groundbreaking column-based file system that enhances performance and improves efficiency in handling large datasets. After extensive testing, we can confirm that this new file system is ready for production. BanyanDB is now production-ready.

In this blog post, we’ll dive deep into the new architecture and the performance improvements observed and provide a step-by-step guide on installing and getting started with BanyanDB v0.6.

Understanding BanyanDB Architecture

BanyanDB is designed as a highly scalable, multi-model database. The architecture of BanyanDB is modular, allowing for flexibility in storage and indexing strategies, which makes it an ideal choice for complex data environments.

Key Features:

Multi-Model Support: Seamlessly handles various data types.
Scalability: Designed to scale horizontally across multiple nodes.
High Performance: Optimized for quick data retrieval and high data ingestion rates.

Data Model

BanyanDB is a multi-model database engineered to support diverse data types, including time series and key-value data. This flexibility is essential for modern APM systems that require versatile data handling capabilities.

BanyanDB Models

Time-Series Data:

BanyanDB manages time-series data, which are data points indexed in time order, typically logged at successive, equally spaced points in time. This makes it ideal for a sequence of discrete-time data. In BanyanDB, you can store time-series data through two structures:

Stream: This type of data is suitable for logging, such as logs, traces, and events. Streams help manage data that are continuously generated and sequentially recorded.
Measure: Designed for ingesting metrics and profiles. Measures are useful for statistical representations over intervals of time.

Key-Value Data:

The key-value model in BanyanDB is a subset of the Property model. Each property is identified by a unique key formatted as <group>/<name>/<id>, which acts as a primary key for retrieving data. This key is immutable once set, ensuring data consistency and integrity.

Properties consist of several key-value pairs, referred to as Tags. You can dynamically add, update, or drop tags based on the tag’s key, offering flexibility in managing and storing data. For example, SkyWalking UI templates utilize this model to store configuration data efficiently.

Clustering in BanyanDB

BanyanDB’s architecture not only ensures efficient data management and high availability but also emphasizes scalability across its clustered environment. The system includes three distinct node types, each capable of scaling independently to meet varying workload demands.

Data Nodes

Data Nodes are central to storing and managing all raw time series data, metadata, and indexed data. Operating under a shared-nothing architecture, these nodes do not communicate directly with each other nor share any data, enhancing cluster availability and simplifying maintenance and scaling. This design prioritizes availability, allowing the system to remain operational for data ingestion and querying even if some nodes are temporarily unavailable.

Meta Nodes

Implemented using etcd, Meta Nodes manage the overarching cluster metadata and ensure consistency across the system. They maintain a global view of the node states and database schemas, facilitating coordinated operations within the cluster.

Liaison Nodes

Liaison Nodes serve as the communication bridge, routing queries and data to the appropriate Data Nodes. They handle security functions like authentication and TTL enforcement, and manage distributed query execution to optimize performance. They play a crucial role in maintaining service availability; as long as at least one Data Node is operational, Liaison Nodes can continue to serve queries. In scenarios where some Data Nodes are unavailable, Liaison Nodes reroute traffic to the remaining healthy nodes, which may lead to increased resource use on these nodes.

BanyanDB Cluster

Communication

Meta Nodes synchronize metadata across the cluster.
Data Nodes interact with Meta Nodes to update and fetch metadata.
Liaison Nodes route data to Data Nodes and coordinate query processes, leveraging metadata from Meta Nodes for efficient distribution and execution.

Scalability and Fault Tolerance

Each node type within the BanyanDB cluster can be scaled independently based on the specific needs of the deployment:

Scaling Data Nodes increases data handling capacity and improves performance under heavier loads.
Scaling Meta Nodes enhances the management capabilities and resiliency of cluster metadata operations. The number of such nodes should be odd.
Scaling Liaison Nodes improves the throughput of query processing and data routing capabilities.

This flexibility allows BanyanDB to adapt to changes in demand without compromising performance or availability. If some components become temporarily unavailable, the system is designed to continue operations, prioritizing availability over strict consistency. However, during such events, if the active nodes do not have sufficient resources to handle the current workload, users may experience delays or failures in data ingestion and query processing.

Installation On Kubernetes

To install BanyanDB on Kubernetes, you can use our Helm chart, which simplifies the deployment process. You can find detailed installation instructions in our official documentation.

This step-by-step guide assumes you have a basic understanding of Kubernetes and Helm, the package manager for Kubernetes. If you’re new to Helm, you might want to familiarize yourself with Helm basics before proceeding.

Prerequisites

Before we begin, ensure you have the following:

A Kubernetes Cluster: You can use Minikube for a local setup, or any cloud provider like AWS, GCP, or Azure that supports Kubernetes.
Helm 3: Ensure Helm 3 is installed and configured on your machine. You can download it from Helm’s official website.

Step 1: Configure Helm to Use OCI

Since the BanyanDB Helm chart is hosted as an OCI chart in Docker Hub, you need to ensure your Helm is configured to handle OCI artifacts.

helm registry login registry-1.docker.io

You will be prompted to enter your Docker Hub username and password. This step is necessary to pull Helm charts from Docker Hub.

Step 2: Setup Env Variables

Next, set up the environment variables for the SkyWalking release version, name, and namespace. These variables will be used in subsequent commands.

export SKYWALKING_RELEASE_VERSION=4.6.0
export SKYWALKING_RELEASE_NAME=skywalking
export SKYWALKING_RELEASE_NAMESPACE=default

Step 3: Install BanyanDB+SkyWalking Using Helm

helm install "${SKYWALKING_RELEASE_NAME}" \
  oci://registry-1.docker.io/apache/skywalking-helm \
  --version "${SKYWALKING_RELEASE_VERSION}" \
  -n "${SKYWALKING_RELEASE_NAMESPACE}" \
  --set oap.image.tag=10.0.1 \
  --set oap.storageType=banyandb \
  --set ui.image.tag=10.0.1 \
  --set elasticsearch.enabled=false \
  --set banyandb.enabled=true \
  --set banyandb.image.tag=0.6.1 \
  --set banyandb.standalone.enabled=false \
  --set banyandb.cluster.enabled=true \
  --set banyandb.etcd.enabled=true

This command will deploy the SkyWalking OAP cluster and BanyanDB cluster to your Kubernetes environment.

Step 4: Verify the Installation

Check the status of the pods to ensure they are running properly:

kubectl get pods -l release=skywalking

You should see the following pods in a Running or Completed state if everything is configured correctly.

Pods in BanyanDB Cluster

Step 5: Access SkyWalking UI

To access the SkyWalking UI, you can check the service by :

kubectl get svc

You should select the service skywalkin-skywalking-helm-ui to access the UI.

Performance Test

We benchmarked BanyanDB v0.6.1 against Elasticsearch 8.13.2, SkyWalking’s recommended database. The new BanyanDB outperformed Elasticsearch in several key areas, particularly in memory usage and disk space.

Data Generation Tool

For this test, we used a custom data generation tool designed to create data that mimics a typical real-world scenario for trace and metrics data.

Services, Instances, and Endpoints

Total Services: 20
Groups of Services: Each of the 3 generator instances runs two groups, contributing to the total count of services.
Instances per Service: Each service is represented by 20 instances, leading to 400 instances across all services.
Endpoints per Service: Each service instance hosts 100 endpoints.
Total Endpoints: With 20 services, the total number of endpoints is 2000.

Trace and Segment Generation

Trace Generation Rate: Each group of services generates 1000 traces per second, effectively simulating a high-load scenario typical in large-scale microservice environments.
Spans per Trace: Each trace comprises five segments, detailing the simulated interactions between various services and instances.
Total Writes per Second: 2 groups * 3 data-generators * 1000 traces * 5 segments = 30k segments.

The additional query types were designed to represent typical read operations performed in a production environment monitoring microservice architectures by SkyWalking. Each query type targets a different aspect of service data:

1. Service Dashboard Queries

Purpose: Fetch 5 service-level metrics over the last 30 minutes.
Frequency: 1 query per second

2. Top-N List Queries

Purpose: Retrieve the top 5 metrics for 2 specific services over the last 30 minutes.
Frequency: 1 query per second.

3. Segment List Queries

Purpose: Fetch a list of service segments ordered by descending latency within the last 30 minutes.
Frequency: 1 query per second.

4. Trace Detail Queries

Purpose: Retrieve all trace details from the segment list.
Frequency: 2 queries per second.

Setup

Below is a detailed table that outlines the specifications of each component within the clusters, allowing for an easy comparison of hardware resources allocated to each system. This provides a clear and structured comparison of the deployment configurations used for Elasticsearch 8.13.2 and BanyanDB v0.6.1 in our performance tests.

Performance Test Setup

Cluster Configuration Table

Component	System	Quantity	CPU Cores	RAM (GB)	Storage (GB)	Role Description
Master Nodes	Elasticsearch	3	2	6	N/A	Cluster coordination and management
Data Nodes	Elasticsearch	3	4	8	50 (Premium RWO)	Data storage, indexing, and query processing
ETCD Nodes	BanyanDB	3	2	4	N/A	Metadata and cluster state storage
Data Nodes	BanyanDB	3	8	4	50 (Premium RWO)	Data storage and processing
Liaison Nodes	BanyanDB	2	4	4	N/A	Coordination between client applications and data nodes

Result

Here we consolidate the performance test results for Elasticsearch 8.13.2 and BanyanDB v0.6.1, focusing on resource usage comparisons. The results are organized into two tables for better clarity—one detailing CPU and memory usage, and the other focusing on disk-related metrics.

CPU and Memory Usage

System	Mean CPU Usage (cores)	Mean Memory Usage (MB)
Elasticsearch Data	3.2	4147
BanyanDB Data	3.6	738
BanyanDB Liaison	1.9	62

Observations:

CPU Usage: BanyanDB data nodes have slightly higher CPU usage due to their operations on compressing and decompressing data files. However, BanyanDB liaison nodes use significantly less CPU.
Memory Usage: BanyanDB shows markedly lower memory usage for both data and liaison nodes, using nearly 5x less memory than Elasticsearch data nodes, highlighting its efficiency in memory utilization.

Disk Usage, IOPS, and Throughput

Disk Usage

Disk IOPS & Throughput

System	Mean Disk Usage (GB)	IOPS (k)	Disk Throughput (GB/s)
Elasticsearch Data	29.6	115.5	12.8
BanyanDB Data	21.6	21.4	3.3

Observations:

Disk Space Usage: BanyanDB utilizes about 30% less disk space than Elasticsearch, which can translate into lower storage costs.
IOPS and Throughput: BanyanDB’s IOPS and disk throughput are significantly lower, indicating less strain on disk resources. This could be beneficial for reducing operational costs and extending the lifespan of physical storage devices.

Conclusion

The release of BanyanDB v0.6 marks a significant advancement in database technology with its new column-based file system. This version demonstrates substantial improvements in both performance and efficiency, particularly in memory usage and disk space compared to Elasticsearch. BanyanDB’s ability to handle various data types, its scalable architecture, and its high performance in data retrieval and ingestion make it a robust solution for complex data environments. The introduction of a flexible clustering system allows for independent scaling of node types, ensuring adaptability to changing demands without compromising on performance or availability. Overall, BanyanDB v0.6 positions itself as a cost-effective and reliable choice for modern application performance management (APM) systems.

Blog: SkyWalking 10 Release: Service Hierarchy, Kubernetes Network Monitoring by eBPF, BanyanDB, and More

Mon, 13 May 2024 00:00:00 +0000

The Apache SkyWalking team today announced the 10 release. SkyWalking 10 provides a host of groundbreaking features and enhancements. The introduction of Layer and Service Hierarchy streamlines monitoring by organizing services and metrics into distinct layers and providing seamless navigation across them. Leveraging eBPF technology, Kubernetes Network Monitoring delivers granular insights into network traffic, topology, and TCP/HTTP metrics. BanyanDB emerges as a high-performance native storage solution, while expanded monitoring support encompasses Apache RocketMQ, ClickHouse, and Apache ActiveMQ Classic. Support for Multiple Labels Names enhances flexibility in metrics analysis, while enhanced exporting and querying capabilities streamline data dissemination and processing.

This release blog briefly introduces these new features and enhancements as well as some other notable changes.

Layer and Service Hierarchy

Layer concept was introduced in SkyWalking 9.0.0, it represents an abstract framework in computer science, such as Operating System(OS_LINUX layer), Kubernetes(k8s layer). It organizes services and metrics into different layers based on their roles and responsibilities in the system. SkyWalking provides a suite of monitoring and diagnostic tools for each layer, but there is a gap between the layers, which can not easily bridge the data across different layers.

In SkyWalking 10, SkyWalking provides new abilities to jump/connect across different layers and provide a seamless monitoring experience for users.

Layer Jump

In the topology graph, users can click on a service node to jump to the dashboard of the service in another layer. The following figures show the jump from the GENERAL layer service topology to the VIRTUAL_DATABASE service layer dashboard by clicking the topology node. Figure 1: Layer Jump

Figure 2: Layer jump Dashboard

Service Hierarchy

SkyWalking 10 introduces a new concept called Service Hierarchy, which defines the relationships of existing logically same services in various layers. OAP will detect the services from different layers, and try to build the connections. Users can click the Hierarchy Services in any layer’s service topology node or service dashboard to get the Hierarchy Topology. In this topology graph, users can see the relationships between the services in different layers and the summary of the metrics and also can jump to the service dashboard in the layer. When a service occurs performance issue, users can easily analyze the metrics from different layers and track down the root cause:

The examples of the Service Hierarchy relationships:

The application song deployed in the Kubernetes cluster with SkyWalking agent and Service Mesh at the same time. So the application song across the GENERAL, MESH, MESH_DP and K8S_SERVICE layers which could be monitored by SkyWalking, the Service Hierarchy topology as below: Figure 3: Service Hierarchy Agent With K8s Service And Mesh With K8s Service.

And can also have the Service Instance Hierarchy topology to get the single instance status across the layers as below: Figure 4: Instance Hierarchy Agent With K8s Service(Pod)
The PostgreSQL database psql deployed in the Kubernetes cluster and used by the application song. So the database psql across the VIRTUAL_DATABASE, POSTGRESQL and K8S_SERVICE layers which could be monitored by SkyWalking, the Service Hierarchy topology as below: Figure 5: Service Hierarchy Agent(Virtual Database) With Real Database And K8s Service

For more supported layers and how to detect the relationships between services in different layers please refer to the Service Hierarchy. how to configure the Service Hierarchy in SkyWalking, please refer to the Service Hierarchy Configuration section.

Monitoring Kubernetes Network Traffic by using eBPF

In the previous version, skyWalking provides Kubernetes (K8s) monitoring from kube-state-metrics and cAdvisor, which can monitor the Kubernetes cluster status and the metrics of the Kubernetes resources.

In SkyWalking 10, by leverage Apache SkyWalking Rover 0.6+, SkyWalking has the ability to monitor the Kubernetes network traffic by using eBPF, which can collect and map access logs from applications in Kubernetes environments. Through these data, SkyWalking can analyze and provide the Service Traffic, Topology, TCP/HTTP level metrics from the Kubernetes aspect.

The following figures show the Topology and TCP Dashboard of the Kubernetes network traffic:

Figure 6: Kubernetes Network Traffic Topology

Figure 7: Kubernetes Network Traffic TCP Dashboard

More details about how to monitor the Kubernetes network traffic by using eBPF, please refer to the Monitoring Kubernetes Network Traffic by using eBPF.

BanyanDB - Native APM Database

BanyanDB 0.6.0 and BanyanDB Java client 0.6.0 are released with SkyWalking 10, As a native storage solution for SkyWalking, BanyanDB is going to be SkyWalking’s next-generation storage solution. This is recommended to use for medium-scale deployments from 0.6 until 1.0.
It has shown high potential performance improvement. Less than 50% CPU usage and 50% memory usage with 40% disk volume compared to Elasticsearch in the same scale.

Apache RocketMQ Server Monitoring

Apache RocketMQ is an open-source distributed messaging and streaming platform, which is widely used in various scenarios including Internet, big data, mobile Internet, IoT, and other fields. SkyWalking provides a basic monitoring dashboard for RocketMQ, which includes the following metrics:

Cluster Metrics: including messages produced/consumed today, total producer/consumer TPS, producer/consumer message size, messages produced/consumed until yesterday, max consumer latency, max commitLog disk ratio, commitLog disk ratio, pull/send threadPool queue head wait time, topic count, and broker count.
Broker Metrics: including produce/consume TPS, producer/consumer message size.
Topic Metrics: including max producer/consumer message size, consumer latency, producer/consumer TPS, producer/consumer offset, producer/consumer message size, consumer group count, and broker count.

The following figure shows the RocketMQ Cluster Metrics dashboard: Figure 8: Apache RocketMQ Server Monitoring

For more metrics and details about the RocketMQ monitoring, please refer to the Apache RocketMQ Server Monitoring,

ClickHouse Server Monitoring

ClickHouse is an open-source column-oriented database management system that allows generating analytical data reports in real-time, it is widely used for online analytical processing (OLAP). ClickHouse monitoring provides monitoring of the metrics 、events and asynchronous metrics of the ClickHouse server, which includes the following parts of metrics:

Server Metrics
Query Metrics
Network Metrics
Insert Metrics
Replica Metrics
MergeTree Metrics
ZooKeeper Metrics
Embedded ClickHouse Keeper Metrics

The following figure shows the ClickHouse Cluster Metrics dashboard: Figure 9: ClickHouse Server Monitoring

For more metrics and details about the ClickHouse monitoring, please refer to the ClickHouse Server Monitoring, and here is a blog that can help for a quick start Monitoring ClickHouse through SkyWalking.

Apache ActiveMQ Server Monitoring

Apache ActiveMQ Classic is a popular and powerful open-source messaging and integration pattern server. SkyWalking provides a basic monitoring dashboard for ActiveMQ, which includes the following metrics:

Cluster Metrics: including memory usage, rates of write/read, and average/max duration of write.
Broker Metrics: including node state, number of connections, number of producers/consumers, and rate of write/read under the broker. Depending on the cluster mode, one cluster may include one or more brokers.
Destination Metrics: including number of producers/consumers, messages in different states, queues, and enqueue duration in a queue/topic.

The following figure shows the ActiveMQ Cluster Metrics dashboard: Figure 10: Apache ActiveMQ Server Monitoring

For more metrics and details about the ActiveMQ monitoring, please refer to the Apache ActiveMQ Server Monitoring, and here is a blog that can help for a quick start Monitoring ActiveMQ through SkyWalking.

Support Multiple Labels Names

Before SkyWalking 10, SkyWalking does not store the labels names in the metrics data, which makes MQE have to use _ as the generic label name, it can’t query the metrics data with multiple labels names.

SkyWalking 10 supports storing the labels names in the metrics data, and MQE can query or calculate the metrics data with multiple labels names. For example: The k8s_cluster_deployment_status metric has labels namespace, deployment and status. If we want to query all deployment metric values with namespace=skywalking-showcase and status=true, we can use the following expression:

k8s_cluster_deployment_status{namespace='skywalking-showcase', status='true'}

related enhancement:

Since Alarm rule configuration had migrated to the MQE in SkyWalking 9.6.0, the alarm rule also supports multiple labels names.
PromeQL service supports multiple labels names query.

Metrics gRPC exporter

SkyWalking 10 enhanced the metrics gPRC exporter, it supports exporting all types of metrics data to the gRPC server.

SkyWalking Native UI Metrics Query Switch to V3 APIs

SkyWalking Native UI metrics query deprecate the V2 APIs, and all migrated to V3 APIs and MQE.

Other Notable Enhancements

Support Java 21 runtime and oap-java21 image for Java 21 runtime.
Remove CLI(swctl) from the image.
More MQE functions and operators supported.
Enhance the native UI and improve the user experience.
Several bugs and CVEs fixed.

Blog: Apache SkyWalking 8.4: Logs, VM Monitoring, and Dynamic Configurations at Agent Side

Fri, 05 Feb 2021 00:00:00 +0000

Origin: Tetrate.io blog

The Apache SkyWalking team today announced the 8.4 release is generally available. This release fills the gap between all previous versions of SkyWalking and the logging domain area. The release also advances SkyWalking’s capabilities for infrastructure observability, starting with virtual machine monitoring.

Background

SkyWalking has historically focused on the tracing and metrics fields of observability. As its features for tracing, metrics and service level monitoring have become more and more powerful and stable, the SkyWalking team has started to explore new scenarios covered by observability. Because service performance is reflected in the logs, and is highly impacted by the infrastructure on which it runs, SkyWalking brings these two fields into the 8.4 release. This release blog briefly introduces the two new features as well as some other notable changes.

Logs

Metrics, tracing, and logging are considered the three pillars of observability [1]. SkyWalking had the full features of metrics and tracing prior to 8.4; today, as 8.4 is released, the last piece of the jigsaw is now in place.

Figure 1: Logs Collected By SkyWalking

Figure 2: Logs Collected By SkyWalking

The Java agent firstly provides SDKs to enhance the widely-used logging frameworks, log4j (1.x and 2.x) [2] and logback [3], and send the logs to the SkyWalking backend (OAP). The latter is able to collect logs from wherever the protocol is implemented. This is not a big deal, but when it comes to the correlation between logs and traces, the traditional solution is to print the trace IDs in the logs, and pick the IDs in the error logs to query the related traces. SkyWalking just simplifies the workflow by correlating the logs and traces natively. Navigating between traces and their related logs is as simple as clicking a button.

Figure 3: Correlation Between Logs and Traces

Infrastructure Monitoring

SkyWalking is known as an application performance monitoring tool. One of the most important factors that impacts the application’s performance is the infrastructure on which the application runs. In the 8.4 release, we added the monitoring metrics of virtual machines into the dashboard.

Figure 4: VM Metrics

Fundamental metrics such as CPU Used, Memory Used, Disk Read / Write and Network Usage are available on the dashboard. And as usual, those metrics are also available to be configured as alarm triggers when needed.

Dynamic Configurations at Agent Side

Dynamic configuration at the backend side has long existed in SkyWalking for several versions. Now, it finally comes to the agent side! Prior to 8.4, you’d have to restart the target services when you modify some configuration items of the agent – for instance, sampling rate (agent side), ignorable endpoint paths, etc. Now, say goodbye to rebooting. Modifying configurations is not the only usage of the dynamic configuration mechanism. The latter gives countless possibilities to the agent side in terms of dynamic behaviours, e.g. enabling / disabling plugins, enabling / disabling the whole agent, etc. Just imagine!

Grouped Service Topology

This enhancement is from the UI. SkyWalking backend supports grouping the services by user-defined dimensions. In a real world use case, the services are usually grouped by business group or department. When a developer opens the topology map, out of hundreds of services, he or she may just want to focus on the services in charge. The grouped service topology comes to the rescue: one can now choose to display only services belonging to a specified group.

Figure 5: Grouped Service Topology

Other Notable Enhancements

Agent: resolves domain names to look up backend service IP addresses.
Backend: meter receiver supports meter analysis language (MAL).
Backend: several CVE fixes.
Backend: supports Envoy {AccessLog,Metrics}Service API V3 and adopts MAL.

Additional Resources

Read more about the SkyWalking 8.4 release highlights.
Get more SkyWalking updates on Twitter.

Blog: Features in SkyWalking 8.2: Browser Side Monitoring; Query Traces by Tags; Meter Analysis Language

Thu, 29 Oct 2020 00:00:00 +0000

Author: Zhenxu Ke, Sheng Wu, Hongtao Gao, and Tevah Platt. tetrate.io
Original link, Tetrate.io blog
Oct. 29th, 2020

Apache SkyWalking, the observability platform, and open-source application performance monitor (APM) project, today announced the general availability of its 8.2 release. The release extends Apache SkyWalking’s functionalities and monitoring boundary to the browser side.

Background

SkyWalking is an observability platform and APM tool that works with or without a service mesh, providing automatic instrumentation for microservices, cloud-native and container-based applications. The top-level Apache project is supported by a global community and is used by Alibaba, Huawei, Tencent, Baidu, ByteDance, and scores of others.

Browser side monitoring

APM helps SRE and Engineering teams to diagnose system failures, or optimize the systems before they become intolerably slow. But is it enough to always make the users happy?

In 8.2.0, SkyWalking extends its monitoring boundary to the browser side, e.g., Chrome, or the network between Chrome and the backend service, or the codes running in the browser. With this, not only can we monitor the backend services and requests sent by the browser as usual, but also the front end rendering speed, error logs, etc., which are the most efficient metrics for capturing the experiences of our end users. (This does not currently extend to IoT devices, but this feature moves SkyWalking a step in that direction).

What’s more, SkyWalking browser monitoring also provides data about how the users use products, such as PV(page views), UV(unique visitors), top N PV(page views), etc., which can give a product team clues for optimizing their products.

Query traces by tags

In SkyWalking’s Span data model, there are many important fields that are already indexed and can be queried by the users, but for the sake of performance, querying by Span tags was not supported until now. In SkyWalking 8.2.0, we allow users to query traces by specified tags, which is extremely useful. For example, SRE engineers running tests on the product environment can tag the synthetic traffic and query by this tag later.

Meter Analysis Language

In 8.2.0, the meter system provides a functional analysis language called MAL(Meter Analysis Language) that allows users to analyze and aggregate meter data in the OAP streaming system. The result of an expression can be ingested by either the agent analyzer or OpenTelemetry/Prometheus analyzer.

Composite Alert Rules

Alerting is a good way to discover system failures in time. A common problem is that we configure too many triggers just to avoid missing any possible issue. Nobody likes to be woken up by alert messages at midnight, only to find out that the trigger is too sensitive. These kinds of alerts become noisy and don’t help at all.

In 8.2.0, users can now configure composite alert rules, where composite rules take multiple metrics dimensions into account. With composite alert rules, we can leverage as many metrics as needed to more accurately determine whether there’s a real problem or just an occasional glitch.

Common scenarios like successful rate < 90% but there are only 1~2 requests can now be resolved by a composite rule, such as traffic(calls per minute) > n && successful rate < m%.

Other Notable Enhancements

The agent toolkit exposes some APIs for users to send customizable metrics.
The agent exclude_plugins allows you to exclude some plugins; mount enables you to load a new set of plugins.
More than 10 new plugins have been contributed to the agent.
The alert system natively supports sending alert messages to Slack, WeChat, DingTalk.

Additional Resources

Read more about the SkyWalking 8.2 release highlights.
Get more SkyWalking updates on Twitter.

Blog: Features in SkyWalking 8.1: SpringSleuth metrics, endpoint dependency detection, Kafka transport traces and metrics

Mon, 03 Aug 2020 00:00:00 +0000

Author: Sheng Wu, Hongtao Gao, and Tevah Platt(Tetrate)
Original link, Tetrate.io blog

Apache SkyWalking, the observability platform, and open-source application performance monitor (APM) project, today announced the general availability of its 8.1 release that extends its functionalities and provides a transport layer to maintain the lightweight of the platform that observes data continuously.

Background

Transport traces

For a long time, SkyWalking has used gRPC and HTTP to transport traces, metrics, and logs. They provide good performance and are quite lightweight, but people kept asking about the MQ as a transport layer because they want to keep the observability data continuously as much as possible. From SkyWalking’s perspective, the MQ based transport layer consumes more resources required in the deployment and the complexity of deployment and maintenance but brings more powerful throughput capacity between the agent and backend.

In 8.1.0, SkyWalking officially provides the typical MQ implementation, Kafka, to transport all observability data, including traces, metrics, logs, and profiling data. At the same time, the backend can support traditional gRPC and HTTP receivers, with the new Kafka consumer at the same time. Different users could choose the transport layer(s) according to their own requirements. Also, by referring to this implementation, the community could contribute various transport plugins for Apache Pulsar, RabbitMQ.

Automatic endpoint dependencies detection

The 8.1 SkyWalking release offers automatic detection of endpoint dependencies. SkyWalking has long offered automatic endpoint detection, but endpoint dependencies, including upstream and downstream endpoints, are critical for Ops and SRE teams’ performance analysis. The APM system is expected to detect the relationships powered by the distributed tracing. While SkyWalking has been designed to include this important information at the beginning the latest 8.1 release offers a cool visualization about the dependency and metrics between dependent endpoints. It provides a new drill-down angle from the topology. Once you have the performance issue from the service level, you could check on instance and endpoint perspectives:

SpringSleuth metrics detection

In the Java field, the Spring ecosystem is one of the most widely used. Micrometer, the metrics API lib included in the Spring Boot 2.0, is now adopted by SkyWalking’s native meter system APIs and agent. For applications using Micrometer with the SkyWalking agent installed, all Micrometer collected metrics could then be shipped into SkyWalking OAP. With some configurations in the OAP and UI, all metrics are analyzed and visualized in the SkyWalking UI, with all other metrics detected by SkyWalking agents automatically.

Notable enhancements

The Java agent core is enhanced in this release. It could work better in the concurrency class loader case and is more compatible with another agent solution, such as Alibaba’s Arthas.

With the logic endpoint supported, the local span can be analyzed to get metrics. One span could carry the raw data of more than one endpoint’s performance.
GraphQL, InfluxDB Java Client, and Quasar fiber libs are supported to be observed automatically.
Kubernetes Configmap can now for the first time be used as the dynamic configuration center– a more cloud-native solution for k8s deployment environments.
OAP supports health checks, especially including the storage health status. If the storage (e.g., ElasticSearch) is not available, you could get the unhealth status with explicit reasons through the health status query.
Opencensus receiver supports ingesting OpenTelemetry/OpenCensus agent metrics by meter-system.

Additional resources

Read more about the SkyWalking 8.1 release highlights.
Read more about SkyWalking from Tetrate on our blog.
Get more SkyWalking updates on Twitter.
Sign up to hear more about SkyWalking and observability from Tetrate.

Blog: SkyWalking v6 is Service Mesh ready

Wed, 05 Dec 2018 00:00:00 +0000

Original link, Tetrate.io blog

Context

The integration of SkyWalking and Istio Service Mesh yields an essential open-source tool for resolving the chaos created by the proliferation of siloed, cloud-based services.

Apache SkyWalking is an open, modern performance management tool for distributed services, designed especially for microservices, cloud native and container-based (Docker, K8s, Mesos) architectures. We at Tetrate believe it is going to be an important project for understanding the performance of microservices. The recently released v6 integrates with Istio Service Mesh and focuses on metrics and tracing. It natively understands the most common language runtimes (Java, .Net, and NodeJS). With its new core code, SkyWalking v6 also supports Istrio telemetry data formats, providing consistent analysis, persistence, and visualization.

SkyWalking has evolved into an Observability Analysis Platform that enables observation and monitoring of hundreds of services all at once. It promises solutions for some of the trickiest problems faced by system administrators using complex arrays of abundant services: Identifying why and where a request is slow, distinguishing normal from deviant system performance, comparing apples-to-apples metrics across apps regardless of programming language, and attaining a complete and meaningful view of performance.

SkyWalking History

Launched in China by Wu Sheng in 2015, SkyWalking started as just a distributed tracing system, like Zipkin, but with auto instrumentation from a Java agent. This enabled JVM users to see distributed traces without any change to their source code. In the last two years, it has been used for research and production by more than 50 companies. With its expanded capabilities, we expect to see it adopted more globally.

What’s new

Service Mesh Integration

Istio has picked up a lot of steam as the framework of choice for distributed services. Based on all the interest in the Istio project, and community feedback, some SkyWalking (P)PMC members decided to integrate with Istio Service Mesh to move SkyWalking to a higher level.

So now you can use Skywalking to get metrics and understand the topology of your applications. This works not just for Java, .NET and Node using our language agents, but also for microservices running under the Istio service mesh. You can get a full topology of both kinds of applications.

Observability analysis platform

With its roots in tracing, SkyWalking is now transitioning into an open-standards based Observability Analysis Platform, which means the following:

It can accept different kinds and formats of telemetry data from mesh like Istio telemetry.
Its agents support various popular software technologies and frameworks like Tomcat, Spring, Kafka. The whole supported framework list is here.
It can accept data from other compliant sources like Zipkin-formatted traces reported from Zipkin, Jaeger, or OpenCensus clients.

SkyWalking is logically split into four parts: Probes, Platform Backend, Storage and UI:

There are two kinds of probes:

Language agents or SDKs following SkyWalking across-thread propagation formats and trace formats, run in the user’s application process.
The Istio mixer adaptor, which collects telemetry from the Service Mesh.

The platform backend provides gRPC and RESTful HTTP endpoints for all SkyWalking-supported trace and metric telemetry data. For example, you can stream these metrics into an analysis system.

Storage supports multiple implementations such as ElasticSearch, H2 (alpha), MySQL, and Apache ShardingSphere for MySQL Cluster. TiDB will be supported in next release.

SkyWalking’s built-in UI with a GraphQL endpoint for data allows intuitive, customizable integration.

Some examples of SkyWalking’s UI:

Observe a Spring app using the SkyWalking JVM-agent

Observe on Istio without any agent, no matter what langugage the service is written in

See fine-grained metrics like request/Call per Minute, P99/95/90/75/50 latency, avg response time, heatmap

Service dependencies and metrics

Service Focused

At Tetrate, we are focused on discovery, reliability, and security of your running services. This is why we are embracing Skywalking, which makes service performance observable.

Behind this admittedly cool UI, the aggregation logic is very easy to understand, making it easy to customize SkyWalking in its Observability Analysis Language (OAL) script.

We’ll post more about OAL for developers looking to customize SkyWalking, and you can read the official OAL introduction document.

Scripts are based on three core concepts:

Service represents a group of workloads that provide the same behaviours for incoming requests. You can define the service name whether you are using instrument agents or SDKs. Otherwise, SkyWalking uses the name you defined in the underlying platform, such as Istio.
Service Instance Each workload in the Service group is called an instance. Like Pods in Kubernetes, it doesn’t need to be a single OS process. If you are using an instrument agent, an instance does map to one OS process.
Endpoint is a path in a certain service that handles incoming requests, such as HTTP paths or a gRPC service + method. Mesh telemetry and trace data are formatted as source objects (aka scope). These are the input for the aggregation, with the script describing how to aggregate, including input, conditions, and the resulting metric name.

Core Features

The other core features in SkyWalking v6 are:

Service, service instance, endpoint metrics analysis.
Consistent visualization in Service Mesh and no mesh.
Topology discovery, Service dependency analysis.
Distributed tracing.
Slow services and endpoints detected.
Alarms.

Of course, SkyWalking has some more upgrades from v5, such as:

ElasticSearch 6 as storage is supported.
H2 storage implementor is back.
Kubernetes cluster management is provided. You don’t need Zookeeper to keep the backend running in cluster mode.
Totally new alarm core. Easier configuration.
More cloud native style.
MySQL will be supported in the next release.

Please: Test and Provide Feedback!

We would love everyone to try to test our new version. You can find everything you need in our Apache repository,read the document for further details. You can contact the project team through the following channels:

Submit an issue on GitHub repository
Mailing list: dev@skywalking.apache.org . Send to dev-subscribe@kywalking.apache.org to subscribe the mail list.
Gitter
Project twitter

Oh, and one last thing! If you like our project, don’t forget to give us a star on GitHub.

Apache SkyWalking – Release

Blog: Apache SkyWalking 2025 in Review: Making BanyanDB Ready for Production

Storage strategy: saying goodbye to H2

BanyanDB: from 0.8.0 foundations to 0.9.0 production features

SkyWalking APM: BanyanDB becomes the default path

Packaging and deployment ecosystem (Helm)

The rest of the community: agents and tooling kept moving

Looking ahead: possible directions in 2026

Closing

Blog: BanyanDB 0.6 Release: Enhanced Performance and Efficiency

Introduction

Understanding BanyanDB Architecture

Key Features:

Data Model

Time-Series Data:

Key-Value Data:

Clustering in BanyanDB

Data Nodes

Meta Nodes

Liaison Nodes

Communication

Scalability and Fault Tolerance

Installation On Kubernetes

Prerequisites

Step 1: Configure Helm to Use OCI

Step 2: Setup Env Variables

Step 3: Install BanyanDB+SkyWalking Using Helm

Step 4: Verify the Installation

Step 5: Access SkyWalking UI

Performance Test

Data Generation Tool

Services, Instances, and Endpoints

Trace and Segment Generation

1. Service Dashboard Queries

2. Top-N List Queries

3. Segment List Queries

4. Trace Detail Queries

Setup

Cluster Configuration Table

Result

CPU and Memory Usage

Observations:

Disk Usage, IOPS, and Throughput

Observations:

Conclusion

Blog: SkyWalking 10 Release: Service Hierarchy, Kubernetes Network Monitoring by eBPF, BanyanDB, and More

Layer and Service Hierarchy

Layer Jump

Service Hierarchy

Monitoring Kubernetes Network Traffic by using eBPF

BanyanDB - Native APM Database

Apache RocketMQ Server Monitoring

ClickHouse Server Monitoring

Apache ActiveMQ Server Monitoring

Support Multiple Labels Names

Metrics gRPC exporter

SkyWalking Native UI Metrics Query Switch to V3 APIs

Other Notable Enhancements

Blog: Apache SkyWalking 8.4: Logs, VM Monitoring, and Dynamic Configurations at Agent Side

Background

Logs

Infrastructure Monitoring

Dynamic Configurations at Agent Side

Grouped Service Topology

Other Notable Enhancements

Links

Additional Resources

Blog: Features in SkyWalking 8.2: Browser Side Monitoring; Query Traces by Tags; Meter Analysis Language

Background

Browser side monitoring

Query traces by tags

Meter Analysis Language

Composite Alert Rules

Other Notable Enhancements

Additional Resources

Blog: Features in SkyWalking 8.1: SpringSleuth metrics, endpoint dependency detection, Kafka transport traces and metrics

Background

Transport traces

Automatic endpoint dependencies detection

SpringSleuth metrics detection