v6.1.8.3

Release v6.1.8.3

v6.1.8.2

new features

• AutoMetrics, AutoTracing, AutoLogging
  • Added support for SOFARPC protocol
• Grafana
  • Support as a Grafana Tempo data source and display Tracing data on the Tempo page

v6.1.8.1

new features

• AutoMetrics
  • Pre-aggregate eBPF Request Log to supplement tap_side = [client process/server process](https://deepflow.yunshan.net/docs/zh/auto-metrics/metrics-without-instrumentation/#statistical position Description) app performance metrics data for

v6.1.8

new features

• AutoMetrics, AutoTracing, AutoLogging
  • Added support for the ProtobufRPC protocol cluster, the current version supports the KRPC protocol
• Management
  • Add a signal_source tag field to the data table to indicate the signal source. Currently, the three signal sources supported for distinction are: Packet (cBPF), eBPF, OTel
  • deepflow-agent supports running in K8s macvlan/ipvlan CNI environment, and automatically calculates Pod access relationship and performance indicators, detailed configuration method of macvlan environment [reference document](https://deepflow.yunshan.net/docs/zh/ install/advanced-config/agent-advanced-config/#macvlan)
  • Support deepflow-agent to run in K8s Pod without SYS_ADMIN permission, detailed permission requirements [reference document](https://deepflow.yunshan.net/docs/zh/install/overview/#Run permission and kernel requirements )
  • Support configuring deepflow-agent to collect the [maximum length] of eBPF Socket Data (https://github.com/deepflowys/deepflow/blob/main/server/controller/model/agent_group_config_example.yaml#L252) to improve collection performance
  • Support configuration of [resource information] (https://github.com/deepflowys/deepflow/blob/main/server/server.yaml#L28) synchronized to the ingester module in deepflow-server, thereby reducing the ingester module under multiple K8s clusters Resource consumption of the AutoTagging feature

optimization

• Management
  • Reduce the memory overhead of deepflow-agent when watching K8s apiserver through message compression and clipping
  • Optimize the memory consumption of deepflow-server when calculating service Tag through Pod aggregation
  • Optimize the write pressure of deepflow-server on the flow_tag database by means of memory pre-aggregation
  • Managed MySQL service supports connections using non-root users

v6.1.7

new features

• Metrics
  • Mark content_length in OTel as metrics, detailed field mapping reference document
  • Request Log increased session_length metric
• Tracing
  • Support parsing the sw8 field in the Dubbo protocol, extracting TraceID, SpanID
    -Event
  • Automatically generate cloud server, K8s Pod add, delete, and change events, and add Grafana Dashboard
• Management
  • Supports configuring the port range for application protocol parsing to reduce the false positive rate. For the configuration method, refer to [l7-protocol-ports configuration item](https://github.com/deepflowys/deepflow/blob/main/server/controller/model/ vtap_group_config_example.go#L816)
  • deepflow-agent supports requesting deepflow-server with a fixed IP address of the load balancer. For the configuration method, refer to [proxy_controller_ip and analyzer_ip fields](https://github.com/deepflowys/deepflow/blob/main/server/controller/model/ vtap_group_config_example.go#L215)
  • Support specifying tap-mode in agent-group-config, so that deepflow-agent can Run in special CNI environment such as macvlan
  • Support configuring kubernetes-cluster-name in deepflow-agent.yaml for specifying K8s cluster name

Optimize

• SQL API
  • When the automatic grouping resource type resource_glX_type is an IP address, directly reuse resource_glX_id to represent the subnet ID, resource_glX to represent the IP address
• Management
  • Reduced database permission requirements when using Managed RDS and ClickHouse
  • Automatically balance the number of deepflow-agents served by each deepflow-server
  • Use OpenTelemetry to monitor the call chain inside deepflow-server

v6.1.6

new features

• AutoLogging
  • Added attribute.http_user_agent and attribute.http_referer fields for HTTP protocol
  • Added attribute.rpc_service field, the value is ServiceName of Dubbo/gRPC
  • Added endpoint field, the value is ServiceName/MethodName of Dubbo/gRPC
  • Support for marking HTTP2 data conforming to the gRPC protocol specification as gRPC (instead of HTTP2) protocol
• AutoTagging
  • Supports synchronizing the resource information of AWS public cloud, and automatically injects it into the observation data as a tag
  • Supports simultaneous injection of cloud resources and container tags for observation data of public cloud-hosted K8s clusters
• manage
  • Support configuring whether deepflow-agent enables parsing of various application protocols
  • Support configuring the regular expression of Golang/openssl process name for deepflow-agent to collect data through eBPF uprobe
  • Supports deepflow-agent standalone mode, where Flow Log and Request Log are written to local log files
  • Support i18n, default display is English

Optimize

• AutoLogging
  • Match SQL keywords to reduce the false positive rate of MySQL and PostgreSQL protocol identification
• Grafana
  • Display the SQL query statement in the Query Editor of the Panel to help developers understand how to call the API
  • Optimized the display of empty field information in Distributed Tracing flame graph
• manage
  • Optimize the traffic between deepflow-server and clickhouse, and preferentially write to the clickhouse Pod on the same node
  • Support to compress OTel Span data received by deepflow-agent, the bandwidth consumption when sending to deepflow-server can be reduced by about 7 times

v6.1.5

new features

• AutoMetrics, AutoTracing, AutoLogging
  • Support collection of PostgreSQL performance indicators and access logs, and associate them with distributed tracing
  • Support to collect HTTPS performance indicators and access logs using openssl library, and correlate them to distributed tracing
• Integration
  • deepflow-server supports RemoteRead interface for Prometheus
  • deepflow-agent supports skipping otel-collector to receive OpenTelemetry data directly
  • The query statement of Grafana Variable supports the use of custom variables and built-in variables, [see documentation](https://deepflow.yunshan.net/docs/zh/server-integration/query/sql/#use-tag-self Name filtering), usage scenarios include:
    • filter the value range of the variable pod with the currently selected value of the variable pod_cluster
    • Use the input content of the variable ingress_wildcard to change the value range of the variable ingress
    • Use the current values ​​of the built-in variables $__from and $__to to improve the query speed of the variable value range
  • Add two zero-intrusive observability dashboards in Grafana: K8s Ingresss, SQL Monitoring
• SQL API
  • string_enum and int_enum types of Tag support using Enum() function to translate Value into Name for query filtering and result return
  • Support SELECT tags/attributes/labels to query all tag.X/attribute.X/label.X fields of each row of data without specifying specific field names
• manage
  • Support ClickHouse cold data to use disk (as an alternative to object storage)
  • Support using deepflow-ctl agent rebalance to balance deepflow-agent to new and restored deepflow-server

Optimize

-AutoLogging

• Sort out the application protocol analysis process and lower the threshold of Add support for more application protocols
• AutoTagging
  • Added deepflow-ctl cloud info command to debug the resource information synchronized from the cloud platform API
• SmartEncoding
  • Recycle the tag encoding value of deleted resources to improve compression ratio and query speed
• SQL API
  • Optimize the display_name of the enumeration value of the server_port Tag field, including the corresponding int value to avoid unclear meaning
• manage
  • deepflow-server is modified to use Deployment Controller to deploy, Please pay attention to update helm chart when upgrading
  • deepflow-agent supports running in unprivileged mode. For specific permission requirements, please refer to [Reference Documentation](https://deepflow.yunshan.net/docs/zh/install/overview/#Running permissions and kernel requirements)
  • Optimize the mapping relationship between Prometheus metrics and DeepFlow Table, each Metrics corresponds to a Table

v6.1.4

1.1. New Features

• AutoTagging
  • Supports synchronizing the resource information of Tencent's public cloud, and automatically injects it into the observation data as a label
• AutoTracing
  • Supports associated application spans and network spans in environments where eBPF cannot run, eliminating tracking blind spots
• SQL API
  • show tags adds fields that return map types, such as labels, attributes, tags
  • show tag values Added limit, offset, like parameters
• Production environment deployment
  • Supports using managed ClickHouse and MySQL
  • When deepflow-agent has not completed registration, it supports configuration issued by deepflow-server
• Grafana
  • Added DeepFlow self-monitoring Dashboard

1.2. Optimization

• SQL API
  • Metrics data in OTel Span can be returned via show metrics API
• system capability
  • Support deepflow-server master election without relying on sidecar
  • deepflow-server supports backward compatibility with deepflow-agent
  • By default, it is synchronized with the NTP server of the container node where deepflow-server is located
  • -v output of normalized process

v6.1.3

Application

AutoMetrics
New indicators: client waiting delay, number of SYN packets, number of SYN-ACK packets, number of SYN retransmission packets, number of SYN-ACK retransmission packets
AutoTracing
Support to associate eBPF uprobe Span with cBPF Span, OTel Span and display it in the trace flame graph
AutoLogging
Support using eBPF uprobe to collect Golang HTTP2, HTTP2_TLS calls
Support collecting Golang process uprobe data with standard symbol table trimmed (Golang >= 1.13 and < 1.18)
AutoTagging
For K8s nodes that are not associated with a cloud server, the cloud server label is automatically generated
Supports synchronizing resource information of Huawei public cloud
Querier SQL API
Fields after GROUP BY are returned automatically, no need to explicitly declare after SELECT
Grafana
Added thumbnail display for DeepFlow Topo and DeepFlow AppTracing
Optimized the Span Tip in the tracing flame graph, showing the time-consuming ratio of the Span itself

System

The time to wait for the agent to come online at the first deployment was optimized from 7 minutes to 4 minutes
Access to deepflow-server and clickhouse in the same K8s cluster no longer uses NodeIP
deepflow-server uses externalTrafficPolicy=Cluster by default to avoid the unavailability of the externalTrafficPolicy=Local function of kube-proxy in some environments and some CNI compatibility issues. It can be manually changed to Local to optimize cross-cluster traffic
deepflow-server adds ext-metrics-ttl, flow-metrics-ttl, flow-log-ttl configuration parameters to initialize data retention time
deepflow-agent supports writing l4_flow_log and l7_flow_log to local files
deepflow-agent removes dependencies on libbpf

v6.1.2

V6.1.2 is the second version after open source, we officially started the rhythm of a small version every two weeks.

Application

• AutoMetrics, AutoLogging
  • MySQL adds COM_STMT_PREPARE, COM_STMT_EXECUTE, COM_STMT_FETCH, COM_STMT_CLOSE command parsing capability
  • Support collecting MQTT 3.1 call logs and performance metrics
• SQL API
  • Support SELECT labels to get all custom label columns

System

• deepflow-agent

  • Added the Inactive IP Indicator Data configuration item. After this configuration is turned off, IPs that do not reply to traffic will be aggregated
  • Use BPF pre-filtered acquisition interface to improve acquisition performance
  • Provide deb installation package

• deepflow-ctl

  • agent-group-config update no longer needs to specify the agent-group-id parameter

• hotfix: #784