Development Build

Commits

• deaeece: fix missing commit sha in sbom-action (#954) (Michael Hoff) #954

19.0.0

Breaking

• backslash in filter and many more expressions gets new meaning, breaking filters or rules that use a plain backslash

Features

• make it possible to assign multiple credentials to a single endpoint
• support escaping json dot notation in filter queries, processor fields and special processor syntaxes

Improvements

• improve http endpoint security by fully checking basic auth hashes, and doing that in a time constant manner to not expose secrets
• improve clusterer performance by removing access via dotted fields where possible
• fix several mypy issues

Bugfix

• fix missing examples for processor decoder
• fix calculator silently failing on syntax errors
• raise TimeParserException if an invalid UNIX timestamp is parsed to prevent timestamper from crashing

Details

• fix: refactor & resolve a subset of mypy issues by @mhoff in #936
• fix: examples for decoder by @mhoff in #942
• feat: add benchmark.py + small logprep-ng adaptions by @kaya-david in #939
• feat: allow multiple credentials for input connector by @Pablu23 in #938
• fix: repair changelog and add automated protection by @mhoff in #943
• perf: improve clusterer performance by removing dotted fields by @ppcad in #940
• feat: add support for escaping json dot notation by @mhoff in #937
• fix: bump build dependencies & fix version in image artifact by @mhoff in #945
• fix: handle invalid UNIX timestamp with TimeParserException by @ppcad in #944
• release: 19.0.0 by @Pablu23 in #946

Full Changelog: v18.1.0...v19.0.0

18.1.0

Features

• add uv as dependency management, including uv.lock
• allow configuration (and auto-creation) of service accounts in helm chart
• add new drop_empty flag to allow the string_splitter to drop resulting fields that would be empty (e.g. whitespace)
• generic_resolver now handles all FieldValue types (including None)

Improvements

• simplify Dockerfile and remove docker build support for LOGPREP_VERSION
• pytest.param now works with test_cases document generation

Bugfix

• generic_resolver now follows yaml standard and accepts a list instead of relying on the ordering of a dict
• generic_resolver now properly handles falsy values in resolve_list and resolve_from_file
• decoder errors are handled properly as warnings instead of causing pipeline failures

Details

• refactor: switch to uv with lockfile, decouple PyPI and optimize Docker builds by @kaya-david in #914
• build: add CI job to verify uv.lock consistency by @kaya-david in #927
• feat: make mapping of Generic Resolver yaml compliant by @Pablu23 in #928
• feat: add service account to chart by @Pablu23 in #931
• test: cover empty http list and empty line handling in list_comparison by @kaya-david in #930
• fix: properly handle DecoderError to avoid pipeline failures by @mhoff in #932
• fix: properly handle falsy resolved values in generic_resolver by @mhoff in #933
• fix: splitting elements with one item by @ekneg54 in #924
• fix: dissector curly braces problem by @ekneg54 in #921
• release: 18.1.0 by @kaya-david in #935

Full Changelog: v18.0.1...v18.1.0

logprep-15.1.0

Logprep helm chart

18.0.1

Features

• headers from incoming http requests can now be copied into events via copy_headers_to_log config in http input, collect_meta will be deprecated in the future
• add new decoder processor to decode values from event field, starting with json, base64, clf (see: https://en.wikipedia.org/wiki/Common_Log_Format), nginx parser for kubernetes ingress, syslog_rfc3164, syslog_rfc3164_local, syslog_rfc5324, logfmt, cri, docker, decolorize (removing color codes in logs)

Improvements

• use follow-imports=silent (instead of skip) to perform more strict type checking
• add docs on how to perform memory profiling
• make the pipeline example work on MacOS (reduce error queue size)
• clean up scheduled jobs and other resources when shutting down components
• fix several minor mypy issues and improve static typing

Bugfix

• fix incorrect default-logger lookup by consistently resolving defaults from DEFAULT_LOG_CONFIG["loggers"]
• fix a possible race condition in the geoip_enricher
• fix possible memory leaks in configuration refresh when processors set up scheduled jobs which were not cleaned up

Details

• test: add coverage for individual logger levels overriding global set… by @kaya-david in #894
• predetector cached field tests by @ppcad in #910
• feat: add acceptance tests for HTTP input metadata collection by @kaya-david in #911
• allow custom http headers to be copied into logs by @Pablu23 in #912
• bug test if config changes on pipeline crash by @Pablu23 in #916
• field_manager can not write to deleted fields by @ekneg54 in #917
• add decoder processor by @ekneg54 in #913
• docs/fix: add how-to on memory profiling and several fixes & improvements by @mhoff in #915
• add some common log formats to the decoder processor by @ekneg54 in #920
• docs: add a pull_request_template by @mhoff in #897
• release: 18.0.1 by @Pablu23 in #925

Full Changelog: v18.0.0...v18.0.1

18.0.0

Breaking

• pre-detector events now also include host.name if the field value is None

Features

• add support for python 3.14
• allow pre-detector to copy a configurable list of fields from log to detection event
• list comparison processor can now also match fields that contain lists in documents
• add network comparison processor that can match IPs with networks in CIDR notation

Improvements

• add workflow to partially run & check the compose example
• add clarification to config_refresh_interval docstring about potential delay under high system load and non-strict timing behavior
• mypy checks in the pull request workflow are now applied to the same directories as in the main workflow
• update codecov-action from v2 to v5
• add token for codecov workflow

Bugfix

• fix opensearch output not respecting thread_count config parameter
• fix docker-compose and k8s example setups
• fix handling of non-string values (e.g. int) as replacement argument for generic_resolver
• fix documentation for generic_resolver rule append_to_list -> merge_with_target option
• fix grokker using a fixed directory for downloaded patterns, potentially leading to conflicts between processes
• fix a bug in the pre_detector that could lead to host.name of previous events being copied into pre-detections of new events

Details

• Doc new runner notebook by @kaya-david in #889
• docs: clarify config_refresh_interval timing under high load by @kaya-david in #893
• Make auto-rule tester only require target_rule_idx if ambiguous by @ppcad in #887
• Refreshable http getters by @ppcad in #882
• fix: replace bitnami images with functional alternatives by @Pablu23 in #891
• fix: resolve mypy issues by @ppcad in #899
• fix: use thread_count in opensearch output by @mhoff in #900
• feat: add workflow to partially run & check the compose example by @mhoff in #898
• Make list comparison accept lists in documents and support matching networks by @ppcad in #885
• feat: upgrade to Python 3.14 and drop support for 3.10 by @Pablu23 in #895
• fix generic_resolve resolve int value by @ekneg54 in #901
• fix: repair automated check for the compose configuration by @mhoff in #902
• Fix generic resolver documentation by @ekneg54 in #903
• feat: allow pre-detector to copy extra fields from log to detection event by @mhoff in #896
• feat: make code coverage checks/reports via codecov stable by @ppcad in #905
• fix: pre-detector not validating against creation_timestamp by @mhoff in #907

New Contributors

• @mhoff made their first contribution in #900

Full Changelog: v17.0.3...v18.0.0

17.0.3

Features

• implement first prototype of ng logprep runner
• ip alerter can now also match fields that contain lists of IPs in documents

Bugfix

• fix error-output not flushing as scheduled

Details

• reintegrate events by @ekneg54 in #875
• Preparation review input connector state by @kaya-david in #876
• revert event reintegration by @ekneg54 in #877
• implement simple sender class by @ekneg54 in #865
• fix kafka tests by @ekneg54 in #881
• Dev implement simple runner by @ekneg54 in #878
• Make dummy input deepcopy repeated documents instead of shallow copy by @ppcad in #883
• Make ip alerter accept ip lists in documents by @ppcad in #884
• clean code by @kaya-david in #886
• Bug error output flush by @Pablu23 in #888

New Contributors

• @Pablu23 made their first contribution in #888

Full Changelog: v17.0.2...v17.0.3

17.0.2

Features

• add clear_event field to add_full_event_to_target_field

Improvements

• add acknowledge() functionality (state change of events and deleting from backlog)
• add event_backlog to the abstract input interface.
• register event in the backlog and return the registered event object.
• make processors handle Event class based objects
• add an EventBacklog class hierarchies
• implement an iterator interface to Input connectors
• make simple connectors handle Event class based objects
• make opensearch_output handle Event class based objects
• deprecate s3_output as it does not fit into new architecture
• deprecate http_output as it does not fit into new architecture
• make confluentkafka_output store Event class based objects
• add new class Pipeline to ng module

Bugfix

• fix auto-rule tester getting stuck due to logging

17.0.1

Improvements

• add ErrorEvent class

• add PseudonymEvent Class

• add SreEvent class

• add LogEvent class

• implement abstract Event class to encapsulate event data, processing state, warnings, and errors

• integrate dotted field handling methods directly into Event, enabling structured field access and manipulation

• support event identity and hashability based on data, allowing usage in sets and as dictionary keys

• implement EventState class to manage the lifecycle of log events

• integrate a finite state machine to control valid state transitions

• add ng packages as namespace in dirs 'unit' and 'logprep' as preparation for new architecture implementation

• add abstract EventMetadata class and KafkaInputMetadata class

• remove ProcessorResult class in favor of LogEvent class

• use LogEvent class in processor base class

Bugfix

• add @timestamp field to error documents

Details

• Dev eventmetadata class implementation by @kaya-david in #823
• Dev eventstate implementation by @kaya-david in #824
• Dev event class implementation by @kaya-david in #825
• Dev logevent class implementation by @kaya-david in #826
• Add Sre Event class by @kmeinerz in #828
• Pseudonym Event class by @kmeinerz in #829
• setup development for refactoring processor classes to use new event class by @ekneg54 in #832
• Add ErrorEvent class by @kmeinerz in #830
• Dev cleanup by @ekneg54 in #834
• Cleanup followup by @kaya-david in #835
• Cleanup followup by @kaya-david in #836
• use new logevent class instead of plain dict by @ekneg54 in #833
• make amides use new logevent class by @ekneg54 in #837
• make calculator use new logevent by @ekneg54 in #838
• Dev use logevent class in concatenator by @kmeinerz in #842
• Dev use logevent class in clusterer by @ekneg54 in #840
• Make datetime extractor use LogEvents by @kmeinerz in #844
• Make deleter use LogEvent class by @kmeinerz in #846
• event backlog implementation by @kaya-david in #845
• Dev use logevent for non extra_data processors by @ekneg54 in #847
• make predetector use new logevents by @ekneg54 in #843
• Dev use logevent pseudonymizer by @ekneg54 in #848
• use logevent selective_extractor by @ekneg54 in #849
• add '@timestamp' to error documents workaround by @kaya-david in #851
• prepare release 17.0.1 by @ekneg54 in #852

New Contributors

• @kaya-david made their first contribution in #823

Full Changelog: v17.0.0...v17.0.1

17.0.0

Breaking

• removed the deprecated kafka generator. The new generator previously available via the kafka2 CLI has been renamed to kafka.

Features

• add replacer processor to replace substrings in fields using a syntax similar to the dissector
• add custom yaml tag !include PATH_TO_YAML_FILE that allows to include other yaml files.
• add custom yaml tags !set_anchor ANCHOR_NAME and !load_anchor ANCHOR_NAME that allow to use anchors across documents inside a file/stream.

Improvements

• ensured that "_test.json" files are not loaded as rules
• introduce new logger Config
• refactor config refresh behavior from logprep.runner to logprep.util.configuration
• refactor config related metrics from logprep.runner to logprep.util.configuration
• added a log message for recovering config refresh mechanic from failing source

Bugfix

• Fixed logging error in _revoke_callback() by adding error handling
• Fixed endless loading in logprep test config
• prevent the auto rule tester from loading rules directly defined inside the config, since they break the auto rule tester and can't have tests anyways
• Fixed typo and broken link in documentation
• Fixed assign_callback error in confluentkafka input
• Fixed error logging in _get_configuration, which caused the github checks to fail
• Resolved mypy errors in BaseProcessorTestCase. by ensuring self.object and self.patchers are not None before accessing attributes.
• Fix domain resolver errors for invalid domains
• Fixed deprecation warnings caused by datetime when using Python >= 3.12
• Fixed timestamp and timezone mismatch issue
• Fixed a bug where config refresh interval was not reset to original interval after recovering from source related failures (i.e. http timeouts)
• Fixed inconsistent generator statistics report during multithreading by making it thread safe

Details

• add replacer processor by @ekneg54 in #672
• fix revoke callback error handling by @kmeinerz in #797
• Add greedy wildcard replacement to replacer by @ppcad in #799
• Fix test config endless load by @kmeinerz in #798
• Prevent auto rule tester from loading rules defined inside config by @ppcad in #809
• small fixes in documentation by @kmeinerz in #810
• Fix kafka input by @kmeinerz in #811
• Ensure "*_test.json" are not loaded as rules by @ppcad in #805
• Fix failing github checks by @kmeinerz in #814
• Add yaml tags !include, !set_anchor and !load_anchor by @ppcad in #804
• Fix domain resolver error, add resolve status and add tests by @ppcad in #808
• Fix datetime deprecation warnings by @ppcad in #815
• remove auto-rule-cli-tests by @ekneg54 in #816
• Fix code quality check of tests by @kmeinerz in #812
• Fix timezone for getting current time via TimeParser by @ppcad in #819
• Removing deprecated kafka generator by @kmeinerz in #820
• Fix-generator-multithreading by @kmeinerz in #821
• Make custom yaml tag loader work with large files by @ppcad in #817
• fix config refresh interval by @ekneg54 in #818
• prepare release 17.0.0 by @ekneg54 in #822

Full Changelog: v16.1.0...v17.0.0