Splunk Heavy Forwarder with NFS shares

[simony]

Hi all,

I have a question regarding log indexing on an NFS share.

The problem we have is, that this NFS share is connected to several systems. Each from these systems writes in the same log which is on the NFS share (clustered application). Because I have not yet found out how 2 or more forwarder shared a logfile pointer, there is the risk of double events in the splunk (Each System indexes the same events). Are there some best practices from splunk side? Is it at all possible to implement such a thing? Or ist the only solution, that the logfiles indexed only by one system in the cluster?

I thank you for your help.

Best Regards,
Yanick

[jethompson_splu]

Hello Yanick,

To address your question in regards to setting up Multiple Heavy Forwarders (or Universal Forwarders) to watch the same Log files on an NFS Mount, you are correct this would cause a "Multiple" ingestions of the same Log, so you would see duplicate entries.

The best solution for this type of setup to monitor your Clustered Applications logs that are being written to an NFS Share would be to install the Splunk Universal Forwarder or Heavy Forwarder, depending on your needs, on the NFS Server. You would then configure the Universal Forwarder/Heavy Forwarder to monitor the Logs that your Cluster Application is writing to on the NFS Share.

The Universal Forwarder is a "Light Weight" version of Splunk that only forwards data to the Indexers for ingestion. The following link will provide information on the Splunk Universal Forwarder:
http://docs.splunk.com/Documentation/Forwarder/6.5.2/Forwarder/Abouttheuniversalforwarder

Now with that being said, alternatively you could have each of the Clustered Application Servers write to local logs and then use the Heavy Forwarder/Universal Forwarder installed on those Servers to monitor the Application Logs. This would provide a means of "tracking" possible issues based on Host Server sending the log data to the Indexers.

Thanks,
Jeff Thompson

[jethompson_splu]

Hello Yanick,

I would like to get some clarification on your use case.

Are you mounting the NFS Share on 2 different Forwarders that are watching the same sets of Logs?

What is the "End Goal" for the configuration that you are wanting to setup?

Are you using a Clustered Configuration or a Distributed Search Configuration?

Thanks,
Jeff Thompson

[simony]

Hi Thompson

Thank you for your quick reply.

In our windows environment we have cluster-based applications (consisting of up to 3 systems), which write all their logs on the NFS share (same logfile). The application runs on one of these systems and can be switched at any time automatically depending on the utilization of the systems. Now we ask ourselves how we monitor such a setup with our Splunk Heavy Forwarders. If we configured a Splunk Heavy Forwarder on each of these systems, which would monitor the same application log files, we would have duplicate data in the Splunk because each server sees the log files on the NFS share. To configure the logfiles only on one of these Heavy Forwarders is also unsightly, because then we have unconsistent configurations over the same application. So it would be nice if there was the possibility that the Heavy Forwarder on all systems monitors the same log files on the NFS share, but the events only occur once in the Splunk. So they somehow share the pointer(fishbucket) for these log files. I hope I could explain it understandly. Now we want to know what methods there are?

To your remaining questions. Yes we are using a clustered configuration with Search and Index Cluster.

Best Regards,
Yanick