splunk duplicate events after forwarder running in a container gets restarted

[rajholla_optum]

We have splunk forwarder running in a docker container and all our workloads which is also running in different containers and writes logs to NFS file mounts on dedicated location.
The problem here is , when container running forwarder restarts it simply sees all file as new and reads them again causing duplicate events.
I assume the problem here is, when forwarder starts in container it becomes new installation of a forwarder.
Can this be solved by persisting forwarder file system(/opt/splunk/splunkforworder/*) ? Or is there any alternative ?

[rfaircloth_splu]

I would also be very concerned the NFS portion of this solution would allow for loss of logs if a close happens without the proper time for flushing to occur. Normally streaming to remote mounts is considered a bad practice. Consider the use of the splunk docker log driver for OS logs and other means of streaming delivery such rsyslog or HEC to avoid the file IO.

[rajholla_optum]

After persisting "/opt/splunk/splunkforwarder" forwarder was able to read the files at specific offset.
Information about fishbucket was very really helpful.

Thank you!

[mattymo · Accepted solution]

if persisting the forwarder is possible (im not a docker guy) then it would solve the issue as Splunk would then remember where it was in each of the files. check out the fishbucket!

https://www.splunk.com/blog/2008/08/14/what-is-this-fishbucket-thing/

[MuS]

Is the Splunk forwarder reading from NFS or writing its logs to NFS?
Check any existing inputs.conf for an option called crcSalt if this is set it can produce duplicate events.

[rajholla_optum]

Thank you for your time !

Forwarder reading log files from NFS.
Forwarder writes everything into container write layer which wont be persisted after restart.
We are not using crcSlat in our inputs.conf