Getting Data In

syslog data missing

Genti
Splunk Employee
Splunk Employee

Folks,

Im trying to troubleshoot an issue where syslog data seems to stop for a couple of days, then pick up again. All on its own.

I have checked metrics.log and there is data coming in.
I have run many searches and have found that the data comes in steadily, and almost constantly.
I have checked that the indexed time is the same as the timestamp splunk gives the events.
What else? - I have checked splunkd.log and made sure that there was no data being blocked, i have done the same on metrics.log

I also have splunked their diag and can confirm that there is no data deletion going on here. The indexes.conf and inputs.conf do not show anything fishy as well. I have also checked to see if there is any data going to the null queue, but see none.

I am in the process of doing some bucket analysis but am awaiting more data from the customer. Any ideas on what else i can look for?
Thanks in advance,
.gz

Tags (1)
1 Solution

Genti
Splunk Employee
Splunk Employee

Issue seems to have been fixed. Not sure if the update to recent version is what fixed it or if they are just better connected to the syslog server.. In anycase, customer seems to be content!

View solution in original post

0 Karma

Genti
Splunk Employee
Splunk Employee

Issue seems to have been fixed. Not sure if the update to recent version is what fixed it or if they are just better connected to the syslog server.. In anycase, customer seems to be content!

0 Karma

bwooden
Splunk Employee
Splunk Employee

I encountered a similar scenario. The above error message was found in splunkd.log. I then learned Splunk was sometimes being started as 'splunkuser' and other times as 'root'. 'root' could access UDP 514, 'splunkuser' could not. I re-directed syslog to a file and monitored file for resolution.

0 Karma

bwooden
Splunk Employee
Splunk Employee

Do you see any "Error binding to socket in UDPInputProcessor: Permission Denied" in splunkd.log?

0 Karma

rotten
Communicator

I would peak at the data coming in with tcpdump or snoop or wireshark just to really see it is what it is expected to be.

0 Karma

Genti
Splunk Employee
Splunk Employee

to confirm that data is not in the system i do a source="udp*"
to confirm that data keeps coming in i check the metrics.log as well as search index=_internal source=metrics.log and see that there are events coming in at a steady, almost constant rate.
Lastly, as i mentioned, when i do a search on the last one, i add _indextime to the fields and see that it is the same as the timestamp that splunk indexes that event (note, here i am talking about index=_internal source="udp*")

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

I suppose my question is, what reason to have to believe that the data ever stops, if metrics show it coming in and searches show continuous data, and you know nothing has been deleted? Where are you not seeing data that you would expect? Also, I have seen a pure auto-timestamping decide that the year of the data is a different year (since syslog doesn't have a year in the timestamp).

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...