Skip to content

Conversation

@BBQigniter
Copy link

@BBQigniter BBQigniter commented Jan 9, 2026

Proposed commit message

This commit should fix a problem we detected using this integration. It seems to stop scraping nginx-ingress-controller logs from Kubernetes clusters with elastic-agents installed. Unfortunately I do not have a lot of knowledge about creating/maintaining integrations and currently have no option to test this change somehow, but by comparing the Kubernetes integration package files with this integration I think I figured out what's missing so that the agent's Filebeat detects container-log rotations correctly and continues to scrape the needed logs.

I think only the fingerprint settings similarly like in the Kubernetes integration are missing in the access- and error-log files - like in

https://github.com/elastic/integrations/blob/main/packages/kubernetes/data_stream/container_logs/manifest.yml#L90-L106
and
https://github.com/elastic/integrations/blob/main/packages/kubernetes/data_stream/container_logs/agent/stream/stream.yml.hbs#L20-L26

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Added configuration options for fingerprinting in the manifest.
Added configuration options for fingerprinting in the manifest.
Add fingerprint settings for container-logs in version 1.14.0.
@BBQigniter BBQigniter requested a review from a team as a code owner January 9, 2026 08:59
Added fingerprint settings for container-logs in version 1.14.0.
@BBQigniter BBQigniter changed the title Nginx controller integration fix [Nginx Ingress Controller Logs] added missing filestream fingerprint settings Jan 9, 2026
@andrewkroh andrewkroh added Integration:nginx_ingress_controller Nginx Ingress Controller Logs Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] labels Jan 9, 2026
@gizas
Copy link
Contributor

gizas commented Jan 12, 2026

I have tested the build locally and all work fine.

Way to test is:

Prerequisites to have a k8s cluster running and also install the elastic-package

  1. Clone the specific branch/pr
  2. Navigate to the folder of this Pr under integrations/integrations/packages/nginx_ingress_controller
  3. Run elastic-package build
elastic-package build
Build the package
2026/01/12 13:51:56  INFO License text found in "/Users/andreasgkizas/integrations/integrations/LICENSE.txt" will be included in package
README.md file rendered: /Users/andreasgkizas/integrations/integrations/packages/nginx_ingress_controller/docs/README.md
Package built: /Users/andreasgkizas/integrations/integrations/build/packages/nginx_ingress_controller-1.14.0.zip
Done
  1. Then deploy the elastic stack locally by running elastic-package stack up -d -v --version=9.2.1
  2. Install Elastic-Agent locally with the Nginx-integration configured

See below the relevant text-boxes that appear in version 1.14.0 of the integration
Screenshot 2026-01-12 at 12 57 47 PM

FYI there is an warning that appears by default in the logs

{"log.level":"warn","@timestamp":"2026-01-12T10:49:27.159Z","message":"1 file is too small to be ingested, files need to be at least 1024 in size for ingestion to start. To change this behaviour set 'prospector.scanner.fingerprint.length' and 'prospector.scanner.fingerprint.offset'. Enable debug logging to see all file names.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.logger":"input.scanner","log.origin":{"file.line":423,"file.name":"filestream/fswatch.go","function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles"},"service.name":"filebeat","filestream_id":"nginx-ingress-controller-access-7a902c467ad8c61e88a0f9dc70af800e3c3cb2b3b9d9d31b2c41b102f527bc8a","ecs.version":"1.6.0"}

You need to update example prospector.scanner.fingerprint.length: 70 in the box that appears above.

Copy link
Contributor

@gizas gizas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gizas
Copy link
Contributor

gizas commented Jan 15, 2026

@andrewkroh have u seen this behaviour again regarding buildkite/integrations ci step?
I can not find this run in buildkite, seems paused and this PR never gets ready to be merged.

Does it have sth to do with the fork and/or with the note above: This branch has not been deployed ? Any help really appreciated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Integration:nginx_ingress_controller Nginx Ingress Controller Logs Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants