I have a question regarding log indexing on an NFS share.
The problem we have is, that this NFS share is connected to several systems. Each from these systems writes in the same log which is on the NFS share (clustered application). Because I have not yet found out how 2 or more forwarder shared a logfile pointer, there is the risk of double events in the splunk (Each System indexes the same events). Are there some best practices from splunk side? Is it at all possible to implement such a thing? Or ist the only solution, that the logfiles indexed only by one system in the cluster?
To address your question in regards to setting up Multiple Heavy Forwarders (or Universal Forwarders) to watch the same Log files on an NFS Mount, you are correct this would cause a "Multiple" ingestions of the same Log, so you would see duplicate entries.
The best solution for this type of setup to monitor your Clustered Applications logs that are being written to an NFS Share would be to install the Splunk Universal Forwarder or Heavy Forwarder, depending on your needs, on the NFS Server. You would then configure the Universal Forwarder/Heavy Forwarder to monitor the Logs that your Cluster Application is writing to on the NFS Share.
Now with that being said, alternatively you could have each of the Clustered Application Servers write to local logs and then use the Heavy Forwarder/Universal Forwarder installed on those Servers to monitor the Application Logs. This would provide a means of "tracking" possible issues based on Host Server sending the log data to the Indexers.
In our windows environment we have cluster-based applications (consisting of up to 3 systems), which write all their logs on the NFS share (same logfile). The application runs on one of these systems and can be switched at any time automatically depending on the utilization of the systems. Now we ask ourselves how we monitor such a setup with our Splunk Heavy Forwarders. If we configured a Splunk Heavy Forwarder on each of these systems, which would monitor the same application log files, we would have duplicate data in the Splunk because each server sees the log files on the NFS share. To configure the logfiles only on one of these Heavy Forwarders is also unsightly, because then we have unconsistent configurations over the same application. So it would be nice if there was the possibility that the Heavy Forwarder on all systems monitors the same log files on the NFS share, but the events only occur once in the Splunk. So they somehow share the pointer(fishbucket) for these log files. I hope I could explain it understandly. Now we want to know what methods there are?
To your remaining questions. Yes we are using a clustered configuration with Search and Index Cluster.