I am currently trying to configure Splunk Universal Forwarders on Windows Workstations. The Universal Forwarder is configured to send security logs directly to our indexer. I have the Windows Add-On installed on the Universal Forwarder, and my inputs.conf file is in the \local\ directory. It is forwarding logs to the indexer as (mostly) intended.
The issue that I am currently experiencing an issue that when the Splunk service restarts on a workstation, it begins forwarding event logs to the indexer that have already been indexed. I'm semi-familiar with what the fishbucket is supposed to do, but it doesn't seem like the indexer is keeping track of my events that have already been indexed 😕
Here's relevant parts from my inputs.conf:
checkpointInterval = 5
disabled = 0
Would greatly appreciate any help you may provide. Thank you!
just to shed some light on this: The problem is that the fishbucket on your forwarder is not keeping track of your logs, not the one on your indexer. Could it be that somehow the fishbucket on your forwarders is getting deleted regularly ?
Or maybe two inputs pointing to the same WinEventLog://Security source?
you can check with btool.
./splunk cmd btool inputs list --debug | grep *://Security
This is a great suggestion, thank you.
I'm nearly positive that there is only one input.conf, I just went through and looked. Furthermore, I cleaned the index so that it was empty and set up a table to monitor for the duplicate events. I'd get one more of the specific event each time I restarted services.
Also, this is installed on Windows, so I am having trouble getting btool to work (I know how to use it on Linux...never thought I'd be one of the 'Actually with Linux it's better because...' guys...). Any pointers on windows?
./splunk cmd btool inputs list --debug | findstr *://Security
I ran btool and everything looks legitimate. I can't for the life of me figure out what's happening. Can you elaborate more on this quote from your original answer?
" The problem is that the fishbucket on your forwarder is not keeping track of your logs, not the one on your indexer. Could it be that somehow the fishbucket on your forwarders is getting deleted regularly ?"
I understand fishbucket is supposed to track this, and I have the 'checkpointInterval' value set to '5', but it doesn't seem to matter. Any idea what's going on?
your fishbucket is stored in $SPLUNK_HOME/var/lib/splunkforwarder/fishbucket
. It keeps track how much of your file/files you want to monitor has been read already
So when you restart your server, it seems to be that this information is then missing, and it reads all the files again. So could it be that you forwarder fishbucket is deleted?
What forwarder version are you running?
Read this for more infos : https://www.splunk.com/blog/2008/08/14/what-is-this-fishbucket-thing.html
Is the fishbucket supposed to be on the Universal Forwarder? I'm starting to think it may be something with the indexer. I originally installed the UF on my test workstation to verify I had the settings correct. As I said earlier, it's forwarding the correct data and indexing it properly. But now I installed it on a second workstation and it's doing the same thing.
They are currently sending directly to the indexer. I think I may switch them to be a heavy forwarder (a new customer requirement). I'm going to switch them both to the heavy forwarder and then have that forward it to the indexer and see if that makes a difference.
I'm currently using Splunk UF 7.2.3 and the indexer I believe is 7.1 (on my list to pilot upgrade soon).