So a few days ago I typo'd the index name in the inputs.conf file on serverA running the universal forwarder, and inadvertently sent the log data to IndexB. We discovered it a few hours ago and since then I have verified multiple times that serverA's inputs is correct now, restarted splunk.
It's been about an hour and this one inputs stanza is still somehow forwarding log data to IndexB instead of IndexA.
I have grepped splunkd.log but all I can see is it parsing the monitored file correctly..nothing about it's destination index.
Any help would be appreciated!
As lag is less than 5s then events are definitely new.
You are absolutely sure that those events are coming from this nodeA and nowhere else? Have you check that those are not coming when you stop UF on nodeA?
Can you double check (with btool) outputs.conf on nodeA ,that it's sending directly to SC instead of HF?
Those two reasons are only ones which comes to my mind for this issue.
BTW: don't run UF as root (this is a security issue). Use instead of e.g. splunk-user and then use setfacl to get access to log files.
I'm pretty sure.. when I query SC I'm using
The events returned also show the source as the asa.log file in question. I have visually checked the other 4 UF servers and the few that have stanzas to monitor an identical path/filename are set to us their own unique index names.
What I do find really odd is, if I tail -f this asa.log file on serverA, it's SUPER busy. Many, many events, however most of those are not hitting indexB.. only some. The log events in SC do contain the IP address of the source feeding asa.log on serverA though.
That might be a difference in raw syslog in the log file, but cisco:asa sourcetype sent to SC? IDK to be honest.
But regardless, I'm trusting the host field in SC for these events as being accurate.
16/09/2022 13:22:04.000 Sep 16 08:22:04 x.x.x.x %ASA-5-713904: IP = y.y.y.y, Received encrypted packet with no matching SA, dropping host: serverA source: /var/log/remote-syslog/asa.log sourcetype: cisco:asa Time _time: 2022-09-16T13:22:04.000+00:00 Default index: indexB linecount: 1 splunk_server: idx-i-xxxxxxxx.splunkcloud.com
I thought about latency.. however I did run
./splunk list inputstatus
And this particular input stanza was showing:
file position = 856704
file size = 856704
percent = 100.00
type = open file
Should that mean that SC is caught up and only receiving new events and if so, any delay should be pretty low.. or at least not on the order of 10+ minutes.
Running your suggestion I get:
root@serverA:/opt/splunkforwarder/bin# ./splunk btool inputs list --debug monitor:///var/log/remote-syslog/asa.log /opt/splunkforwarder/etc/apps/200_syslog_forwarders/local/inputs.conf [monitor:///var/log/remote-syslog/asa.log] /opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864 /opt/splunkforwarder/etc/apps/200_syslog_forwarders/local/inputs.conf disabled = 1 /opt/splunkforwarder/etc/system/default/inputs.conf host = $decideOnStartup /opt/splunkforwarder/etc/apps/200_syslog_forwarders/local/inputs.conf index = indexA /opt/splunkforwarder/etc/apps/200_syslog_forwarders/local/inputs.conf sourcetype = cisco:asa
Which is correct.. And as I type this this stanza has been disabled for almost 15 minutes, my query in SC is still showing events from serverA host coming in, it's boggling!
"./splunk list inputstatus" shows only what UF has done, it's not told that events are already in indexers unless you have use useACK on outputs conf.
You should check lag on SH side something like
index=<your index(es)> source=<your source> host=<your host> | eval lag = _indextime - _time | stats avg(lag) as aLag max(lag) as mLag | eval aLagHuman = tostring(aLag, "duration"), mLagHuman = tostring(mLag, "duration")
Have you any HF on path to SC which can do transforms?
Another reason could be that you have so long lag on receiving events, that those which are now coming to SC are collected and sent before that change?
splunk btool inputs list --debug monitor:< your source file >
on UF to check that you have done change to correct inputs.conf file. Of course if this has managed by DS then you must do the change on DS side.