4.5.0

What's new

• Merge pull request #711 from orb-community/develop [Michal Fiedorowicz]
• fix build by updating json validator version (#709) [Leonardo Parente]
• Merge pull request #707 from orb-community/develop [Michal Fiedorowicz]
• Bump urllib3 from 1.26.11 to 1.26.18 in /automated_tests (#706) [dependabot[bot]]
• Update dependency libraries (#704) [Leonardo Parente]
• Bump requests from 2.28.1 to 2.31.0 in /automated_tests (#705) [dependabot[bot]]
• Bump certifi from 2022.6.15 to 2023.7.22 in /automated_tests (#703) [dependabot[bot]]
• bump version to 4.5.0 (#702) [Michal Fiedorowicz]
• OTLP: Move policy name to attributes (#699) [Leonardo Parente]
• Upgrade pcapplusplus from 22.11 to 23.09 (#701) [Leonardo Parente]
• Upgrade uvw to 3.X and fix/suppress some compiler warnings (#694) [Leonardo Parente]
• Update conan CI version to 1.61.0 and re-enable macos build (#693) [Leonardo Parente]
• Migrate from catch2v2 to catch2v3 (#692) [Leonardo Parente]
• Revert "migrate from catch2v2 to catch2v3 and implement VisorTest helper lib (#689)" (#691) [Shannon Weyrick]
• migrate from catch2v2 to catch2v3 and implement VisorTest helper lib (#689) [Leonardo Parente]
• Simplify and fix TCP netprobe probe (#690) [Shannon Weyrick]
• [ENG-786] • fix: pktvisor-cli for arm64 architecture (#688) [Everton H. Taques]
• Update current dependencies to latest (#687) [Leonardo Parente]
• Update Dockerfile to use bullseye image (#686) [Everton H. Taques]
• Increase policies unit tests and code coverage (#685) [Leonardo Parente]
• upgrade PcapPlusPlus dependency from v22.05 to v22.11 (#663) [Leonardo Parente]
• Upgrade datasketches lib from 3.3.0 to 4.1.0 (#684) [Leonardo Parente]
• Unit Tests Visor Core: cover StreamHandler Class (#682) [Leonardo Parente]

4.4.0

What's new

• Pktvisor release 4.4.0 (#681) [Everton H. Taques]
• disabling macos build on release pipeline (#680) [Everton H. Taques]
• disable macos build temporarily (#679) [Everton H. Taques]
• Ensure TopN _item_key label has proper value before adding opentelemetry datapoint (#678) [Leonardo Parente]
• add autoretry to automated tests (#677) [manrodrigues]
• [Fix] app image dockerfile (#676) [Everton H. Taques]
• [fix] remove armv7 (#674) [Everton H. Taques]
• Add nullptr validation on GeoDB class (#675) [Leonardo Parente]
• [fix] pktvisor report tests [Everton H. Taques]
• [fix] pktvisor report tests [Everton H. Taques]
• Bump develop version (#672) [Everton H. Taques]

4.3.0

New Features

• [OpenTelemetry] Add Opentelemetry Exporter support to push Pktvisor metrics #584 #604 #654 #664 #665
• Add support to Net Handler V2 #615
• Flow: only_directions filter and exclude_asns_from_summarization config #585 #591
• Exact match Qname filter and make improvements on O(1) filtering #588
• Add Netflow support to ipv6 #599
• Add IP ToS support for and ECN to Flow Handler #649 #651
• Add chaining support for Pcap TCP packets #609

Other Changes

• Reduce Histogram Cardinality #595
• Explicit support IPFIX as flow type #605
• Flow - Add support to subnet wildcard summary #613
• Flow - Rename ip_port top metrics #621
• Limit tcp reassembly packets #662
• Do not return prometheus metric if it is empty #669

Bug Fixes

• Fix dns V2 sintax issues and add version to StreamHandler by @leoparente #573
• Fix Dns V2 specialized merge by @leoparente #590 #594
• Fix set_info for Rate Metric by @leoparente #601
• PingProbe release internal timer by @leoparente #606
• Fix asn_exclude in FlowHandler by @leoparente #608
• Fix Dns V2 period shift by @leoparente #614
• Fix Flow Ipv6 subnet summarization by @leoparente #618
• Fix debug build and release pipeline by @etaques #622

New Contributors

@egecetin made their first contribution in #637
@emn made their first contribution in #640

4.2.0

New Features in BETA (interfaces may still change)

• Flow Support (SFLOW/Netflow/IPFIX). How To Configure and Policies Advanced.
• Netprobe support. Docs.
• Histogram Metric #526
• DNS Handler Version 2.0 - focus on dns transactions (docs soon)

New Features

• Support pktvisor build with MSVC (Windows) using VS2019. pktvisord.exe will be included in next releases!
• Add crashpad support for remote crash reports
• Improve pktvisor CI/CD pipeline by adding tests and decreasing time by @etaques (#357) (#446)
• change prom label of handler from 'module' to 'handler' by @weyrick (#455)
• Implement merge_like_handlers feature on Policies (#426). Docs.
• Implement BGP Handler (#376). Docs
• Implement network interface scan and pick default tap - pktvisord 'auto' support (#373) [Leonardo Parente]
• Implement remote tap support (#362) [Leonardo Parente]
• New DNS Handler Filters (qtype, DNSSEC, ECS Geo Not Found).
• Implement tap selector (#330) [Leonardo Parente]
• adapting integration scenarios to run in parallel processes by @manrodrigues (#322) (#435)
• Add RFC Config #307 (#311) [Leonardo Parente]
• Feature/configurable topn count (#308) [Leonardo Parente]
• Add global handler configuration at agent level (#304) [Leonardo Parente]

Full Changelog: v4.1.0...4.2.0

4.1.0

New Features

• Metrics Groups for enabling/disabling metrics in handlers in #212
• Create dnstap_msg_type filter in DNS handler in #216
• Implement CPU/Mem resource tracking application metrics in #230
• Adjust qname aggregation to limit cardinality in #242
• Add support to packet size and throughput to Net handler in #261
• Feature/filter dnstap by hostmask in #217

Other Changes

• Improve TCP timeout with LRU list in #249
• Allow creating policies using JSON in #198
• Cross compilation for ARM architecture in #203 and #215
• Update pcapplusplus version and 3rd party libs in #224 and #231
• Add gdb to docker debug in #234 and #235
• Optimization: improve handler processing with local_thread cached data in #243
• Automated tests for pktvisor policies in #237, #248 and #251
• Add crashpad to debug images in #260 and #265

Bug Fixes

• Fix issue when deleting PCAP policy with BPF filter by @leoparente in #201
• Fix global configuration of period and deep sample for window config by @weyrick in #211
• Add new_event to pcap handler callbacks to allow new bucket creation which fixes an issue when scraping bucket metrics by @leoparente in #227
• Avoid invalid DNS packets in TCP by @leoparente in #262

New Contributors

• @manrodrigues made their first contribution in #237
• @etaques made their first contribution in #265

Full Changelog: v4.0.0...4.1.0

4.0.0

New Features

• New dnstap input module (full analyzer support) #155 and #176
• New sFlow input module (basic analyzer support) #187
• Support for dynamically linked and loaded input and analyzer modules #156 and #179 and #179
• Support chaining in policy handlers which allow execution of an analyzer only if the analyzer before it doesn't filter #169
• clang toolchain builds, creating fully statically linked executables for Linux #168

Backwards Incompatible Changes

• Unique name for prometheus TopN metric #183

The "name" label on Top N metrics in Prometheus output has changed to be domain specific - for example dns_top_qname2{instance="gw",name=".aaplimg.com"} is now dns_top_qname2{instance="gw",qname=".aaplimg.com"} and dns_top_rcode{instance="gw",name="NOERROR"} is now dns_top_rcode{instance="gw",rcode="NOERROR"}. Please update your dashboards accordingly, or use the latest community dashboard.

Other Changes

• DNS qname labels are now lower-cased before being inserted into Top N tables, reducing cardinality and improving aggregation
• optimize: reuse Tap if the same input is used by multiple policies #186
• add __all policy support for prometheus metics #142
• Support to cmd all options through yaml config file #159 and #164
• pktvisor debug/sanitized build #190

Bug Fixes

• Remove prom help from loop, removing redundancy and fixing scrapers by @weyrick in #144
• Remove stopped InputStream and StreamHandlers from Registry managers by @leoparente in #172
• policy threads exception by @weyrick in #178

New Contributors

• @leoparente made their first contribution in #159

Full Changelog: v3.3.0...v4.0.0

3.3.0

New Features

• Implement RFCs #75 and #76 for Taps and Collection Policies. These interfaces should be considered Beta and may change.
• Introduce YAML based configuration files (--config), used for commandline configuration, taps, and policies.
• New DHCP handler module #131
• PcapPlusPlus interface concurrency in #123
• Add new mock traffic generator to pcap input source (set pcap_source config to mock) #134
• Implement DNS filters, giving policies the ability to limit DNS metrics to qname or rcode #78

Other Changes

• --prometheus is now no longer need; Prometheus support is always enabled
• Improved test suite and CI
• Refactor golang CLI

Bug Fixes

• Switch random number generators used by sampling by @weyrick in #110
• Fix Corrade detection and build failures with GCC 11 by @fcelda in #130
• Fix DNS rates in CLI #126

New Contributors

• @rlahnemann made their first contribution in #111
• @iatdaitan made their first contribution in #113
• @fcelda made their first contribution in #130

Full Changelog: v3.2.0...v3.3.0

3.2.1

New Features

• #94 Introduce TLS support to the pktvisor administration webserver. See --tls, --tls-cert, --tls-key command line options
• #50 Introduce new pcap handler module to collect operational metrics from pcap input module. This is now added automatically to default analysis, and introduces new top level result object pcap with metrics tcp_reassembly_errors, os_drops and if_drops. The drop metrics come from libpcap

Other Improvements

• #98 Improved support for handling PcapPlusPlus error messages
• #93 Upgrade to PcapPlusPlus v21.05. Fixes some double parsing of DNS messages.

Bug Fixes

• #83 Improvements to demonize
• #84 Improvements to syslog and file logging
• #92 Increase HTTP time out in CLI
• #100 Handle TCP reassembly errors better

3.2.0

New Features

• Introduce native Prometheus support into pktvisord with --prometheus flag, which will expose Prometheus compatible metrics at /metrics endpoint. Also see --prom-instance
• Add a new docker container for easily collecting and sending Prometheus compatible metrics, see docker hub
• Add a new Grafana dashboard for Prometheus, both to the repo and to Grafana dashboard community
• Begin building and distributing an AppImage (static Linux binary) which includes pktvisord, pktvisor-cli, and pktvisor-pcap
• Ability to deamonize pktvisord with the -d flag
• Ability to send pktvisord logs to either an output file (--log-file), or to syslog (--syslog)

Other Improvements

• CI and build improvements including better use of Conan and automatic dependency installation
• Improved documentation and READMEs

Bug Fixes

• #47 Fix live rates in pktvisor-cli

3.1.0

Summary

• Major refactor of code base and build system to modularize input streams, the metrics subsystem, and stream handlers. Modules may be dynamically or statically linked, and are loadable at runtime. The first modules are the "pcap" input module and the "net" and "dns" stream handlers, which are currently statically linked into pktvisord binary.
• Introduce --admin-api with full control plane functionality to prepare for creating observability policies. The admin API is not yet stable and should be expected to change.
• Update to the latest version of several dependencies such as Apache Data Sketches and PcapPlusPlus
• Begin using Conan package manager for C++ dependencies, and Github actions for CI
• Switch the open source license from Apache to Mozilla Public License

3.0.x Compatibility Notes

• pcap file analysis has been split out of pktvisord and into its own binary pktvisor-pcap
• The command line UI has been renamed from pktvisor to pktvisor-cli. The old name is currently still accepted in the docker container entry.
• The /api/v1/metrics/rates endpoint is being deprecated; it still works but the rates are always 0. Instead, the "live" bucket (bucket number 0) includes rate information under the "live" keys. pktvisor-cli has been updated accordingly, and this change does not affect typical central collection scenarios which only collect bucket number 1.