Client is has a clustered Active-DR setup for their PROD application. At a given time, only one server (node) is active and mounted with common NFS share.
When application switches over to the other node, NFS share (File system mount point) is unmounted from active one and same is mounted on another node.
We have a requirement to configure the Splunk universal forwarder on both the nodes. We can ask the support team to manually stop/start the Splunk forwarder during migration (switch over).
However, not sure how Splunk universal forwarder will behave while reading (tailing) same log file from a different forwarder and indexing in the same index.
The Splunk (Universal) forwarder on the failed-over node will re-read whole file data after switch over. This is the default behavior (all data would be monitored) as that node has never monitored the file before. This is controlled by a property called followTail on the inputs.conf file on forwarder. This is 0 (false) by default means monitoring starts at the beginning of the file. See this (search for followTail) for more details.
One option would be to manually set the followTail in inputs.conf to 1 / true so that monitoring starts at the end of the file (like tail -f). Please note that this is an advanced setting and should be used for temporary purpose only.
Excerpt From inputs.conf
* WARNING: Use of followTail should be considered an advanced administrative
* Treat this setting as an 'action':
* Enable this setting and start the Splunk software.
* Wait enough time for the input to identify the related files.
* Disable the setting and restart.
* DO NOT leave followTail enabled in an ongoing fashion.
* Do not use followTail for rolling log files (log files that get renamed as
they age), or files whose names or paths vary.