Splunk Enterprise

splunk duplicate events after forwarder running in a container gets restarted

rajholla_optum
Engager

We have splunk forwarder running in a docker container and all our workloads which is also running in different containers and writes logs to NFS file mounts on dedicated location.
The problem here is , when container running forwarder restarts it simply sees all file as new and reads them again causing duplicate events.
I assume the problem here is, when forwarder starts in container it becomes new installation of a forwarder.
Can this be solved by persisting forwarder file system(/opt/splunk/splunkforworder/*) ? Or is there any alternative ?

Tags (1)
0 Karma
1 Solution

mattymo
Splunk Employee
Splunk Employee

if persisting the forwarder is possible (im not a docker guy) then it would solve the issue as Splunk would then remember where it was in each of the files. check out the fishbucket!

https://www.splunk.com/blog/2008/08/14/what-is-this-fishbucket-thing/

- MattyMo

View solution in original post

rfaircloth_splu
Splunk Employee
Splunk Employee

I would also be very concerned the NFS portion of this solution would allow for loss of logs if a close happens without the proper time for flushing to occur. Normally streaming to remote mounts is considered a bad practice. Consider the use of the splunk docker log driver for OS logs and other means of streaming delivery such rsyslog or HEC to avoid the file IO.

rajholla_optum
Engager

After persisting "/opt/splunk/splunkforwarder" forwarder was able to read the files at specific offset.
Information about fishbucket was very really helpful.

Thank you!

0 Karma

mattymo
Splunk Employee
Splunk Employee

if persisting the forwarder is possible (im not a docker guy) then it would solve the issue as Splunk would then remember where it was in each of the files. check out the fishbucket!

https://www.splunk.com/blog/2008/08/14/what-is-this-fishbucket-thing/

- MattyMo

MuS
SplunkTrust
SplunkTrust

Is the Splunk forwarder reading from NFS or writing its logs to NFS?
Check any existing inputs.conf for an option called crcSalt if this is set it can produce duplicate events.

0 Karma

rajholla_optum
Engager

Thank you for your time !

Forwarder reading log files from NFS.
Forwarder writes everything into container write layer which wont be persisted after restart.
We are not using crcSlat in our inputs.conf

0 Karma
Get Updates on the Splunk Community!

September Community Champions: A Shoutout to Our Contributors!

As we close the books on another fantastic month, we want to take a moment to celebrate the people who are the ...

Splunk Decoded: Service Maps vs Service Analyzer Tree View vs Flow Maps

It’s Monday morning, and your phone is buzzing with alert escalations – your customer-facing portal is running ...

What’s New in Splunk Observability – September 2025

What's NewWe are excited to announce the latest enhancements to Splunk Observability, designed to help ITOps ...