Splunk Forwarder migration

[mehmettecer]

Hi guys,

Here is my issue:
I have 2 rsyslog servers that are in production in redundant setup. Other servers forward same logs to these servers at the same time. Due to UDP drops, log files are slightly different on both servers.

I have Forwarder setup on only one of them, lets call it Server A, now I need to retire Server A and replace it with Server B.

My question here is, when Server B comes up, will Splunk start indexing all the log files as if it was a new server ? If yes, how can I avoid duplicate indexed data in ? Is there are Splunk best practice for such situation ? I know splunk forwarder remembers where it is was left while monitoring files, is there a way to transfer that monitoring offset data to the new server so once Server B comes up it starts monitoring from where it left off ?

Thanks

[jasonnadeau]

Can you expand on your rsyslog configuration a little bit? My rsyslog creates a directory hierarchy based upon the source IP address for the log message, then creates a new log file per day for each unique IP address.

+/syslog
+-/192.168.1.1
---/2011-10-04.log
+-/192.168.1.10
---/2011-10-04.log
---/2011-10-03.log

If you need to replace Server A with Server B on a scheduled date and time you may want to move the older data out of "view" of splunk monitored directories just before you fire up Splunk on Server B, then you can shutdown Splunk on Server A. At that point the log data will be ingested via Server B, and once you are satisfied you are getting all the required logs you can decommission Server A. If things appear to be going left you can always turn Splunk back on server A and it will pickup logs from where it left off.