syslog data missing

[Genti]

Folks,

Im trying to troubleshoot an issue where syslog data seems to stop for a couple of days, then pick up again. All on its own.

I have checked metrics.log and there is data coming in.
I have run many searches and have found that the data comes in steadily, and almost constantly.
I have checked that the indexed time is the same as the timestamp splunk gives the events.
What else? - I have checked splunkd.log and made sure that there was no data being blocked, i have done the same on metrics.log

I also have splunked their diag and can confirm that there is no data deletion going on here. The indexes.conf and inputs.conf do not show anything fishy as well. I have also checked to see if there is any data going to the null queue, but see none.

I am in the process of doing some bucket analysis but am awaiting more data from the customer. Any ideas on what else i can look for?
Thanks in advance,
.gz

[Genti · Accepted solution]

Issue seems to have been fixed. Not sure if the update to recent version is what fixed it or if they are just better connected to the syslog server.. In anycase, customer seems to be content!

[bwooden]

I encountered a similar scenario. The above error message was found in splunkd.log. I then learned Splunk was sometimes being started as 'splunkuser' and other times as 'root'. 'root' could access UDP 514, 'splunkuser' could not. I re-directed syslog to a file and monitored file for resolution.

[bwooden]

Do you see any "Error binding to socket in UDPInputProcessor: Permission Denied" in splunkd.log?

[rotten]

I would peak at the data coming in with tcpdump or snoop or wireshark just to really see it is what it is expected to be.

[Genti]

to confirm that data is not in the system i do a source="udp*"
to confirm that data keeps coming in i check the metrics.log as well as search index=_internal source=metrics.log and see that there are events coming in at a steady, almost constant rate.
Lastly, as i mentioned, when i do a search on the last one, i add _indextime to the fields and see that it is the same as the timestamp that splunk indexes that event (note, here i am talking about index=_internal source="udp*")

[gkanapathy]

I suppose my question is, what reason to have to believe that the data ever stops, if metrics show it coming in and searches show continuous data, and you know nothing has been deleted? Where are you not seeing data that you would expect? Also, I have seen a pure auto-timestamping decide that the year of the data is a different year (since syslog doesn't have a year in the timestamp).