We have 3 firewalls that are sending their syslogs to a udp port. 2 are showing events, one is not. It's like the events aren't being indexed. I tried sending the 1 firewall we aren't seeing data for to a different port mapped to the same source type to no avail. I do see splunkd listening on that port, and tcpdump shows data from that ip coming in on that port. What further steps may I take to diagnose this?
Thank you in advance!!!
Check splunkd.log on your indexers for any messages related to Timestamp parsing (Look for DateParserVerbose component). Is it possible that timestamps from the suspect firewall come in with a time in the future?
Thank you both for your input! The source is mentioned in metrics.log. ssievert, I see no mention of timestamp parsing 😞
Could you run:
| tstats count, max(_indextime) AS mostrecentindex, max(_time) AS mostrecent where index=<yourindex>, source=<yoursource> | eval mostrecentindex=strftime(mostrecentindex, "%+"), mostrecent = strftime(mostrecent, "%+")
The where clause can have sourcetype/source/index or similar, just narrow it down as appropriate and run that over a very wide time range or all time to determine if your events are going in with invalid timestamps or not....
Returns a count of zero, other two fields are blank, so I am guessing that it is not even being indexed?
Can you run
| tstats count where index=yourindexname by host and see if your source host is listed?
| metadata type=hosts to see if your firewall host is listed at all.
Finally, do you get any results if you run a 30-second windowed real-time search using
I see it in the first two, but not the last. There seems to be a disparity though between last time and recent time..... Much more so than anything else.
OK, so the tstats is showing events from that host in the given index, but when you are searching for data from that host, you are not coming up with anything?
I still suspect something is going on with the timestamps/timezone from that host. Can you compare the system settings for the 3 firewall devices and rule out that they differ in timezone settings?
Already did, timezone is the same across all 3. Time is the same as they are all synced using NTP. Funny, I found my events! Splunk is dating them as being 2 years prior! But I am positive the date on the firewalls are all the same.