topic Re: splunk duplicate events after forwarder running in a container gets restarted in Splunk Enterprise

splunk duplicate events after forwarder running in a container gets restarted

rajholla_optum — Sun, 26 Mar 2017 17:43:06 GMT

We have splunk forwarder running in a docker container and all our workloads which is also running in different containers and writes logs to NFS file mounts on dedicated location.
The problem here is , when container running forwarder restarts it simply sees all file as new and reads them again causing duplicate events.
I assume the problem here is, when forwarder starts in container it becomes new installation of a forwarder.
Can this be solved by persisting forwarder file system(/opt/splunk/splunkforworder/*) ? Or is there any alternative ?

Re: splunk duplicate events after forwarder running in a container gets restarted

MuS — Sun, 26 Mar 2017 19:18:38 GMT

Is the Splunk forwarder reading from NFS or writing its logs to NFS?
Check any existing inputs.conf for an option called crcSalt if this is set it can produce duplicate events.

Re: splunk duplicate events after forwarder running in a container gets restarted

rajholla_optum — Mon, 27 Mar 2017 01:07:20 GMT

Thank you for your time !

Forwarder reading log files from NFS.
Forwarder writes everything into container write layer which wont be persisted after restart.
We are not using crcSlat in our inputs.conf

Re: splunk duplicate events after forwarder running in a container gets restarted

mattymo — Mon, 27 Mar 2017 01:22:47 GMT

if persisting the forwarder is possible (im not a docker guy) then it would solve the issue as Splunk would then remember where it was in each of the files. check out the fishbucket!

https://www.splunk.com/blog/2008/08/14/what-is-this-fishbucket-thing/

Re: splunk duplicate events after forwarder running in a container gets restarted

rajholla_optum — Mon, 27 Mar 2017 17:19:56 GMT

After persisting "/opt/splunk/splunkforwarder" forwarder was able to read the files at specific offset.
Information about fishbucket was very really helpful.

Thank you!

Re: splunk duplicate events after forwarder running in a container gets restarted

rfaircloth_splu — Mon, 27 Mar 2017 17:29:39 GMT

I would also be very concerned the NFS portion of this solution would allow for loss of logs if a close happens without the proper time for flushing to occur. Normally streaming to remote mounts is considered a bad practice. Consider the use of the splunk docker log driver for OS logs and other means of streaming delivery such rsyslog or HEC to avoid the file IO.