Getting Data In
Highlighted

Why isnt't our firewall showing events? We're sending syslogs to a UDP port

Path Finder

Good afternoon,
We have 3 firewalls that are sending their syslogs to a udp port. 2 are showing events, one is not. It's like the events aren't being indexed. I tried sending the 1 firewall we aren't seeing data for to a different port mapped to the same source type to no avail. I do see splunkd listening on that port, and tcpdump shows data from that ip coming in on that port. What further steps may I take to diagnose this?

Thank you in advance!!!

0 Karma
Highlighted

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

Influencer

Check splunkd.log on your indexers for any messages related to Timestamp parsing (Look for DateParserVerbose component). Is it possible that timestamps from the suspect firewall come in with a time in the future?

Highlighted

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

SplunkTrust
SplunkTrust

Also check within your servers metrics.log to see if that source is ever mentioned, the metrics log will mention the 10 busiest sources processed by this particular server...

0 Karma
Highlighted

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

Path Finder

Thank you both for your input! The source is mentioned in metrics.log. ssievert, I see no mention of timestamp parsing 😞

0 Karma
Highlighted

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

SplunkTrust
SplunkTrust

Could you run:

| tstats count, max(_indextime) AS mostrecentindex, max(_time) AS mostrecent where index=<yourindex>, source=<yoursource>
| eval mostrecentindex=strftime(mostrecentindex, "%+"), mostrecent = strftime(mostrecent, "%+")

The where clause can have sourcetype/source/index or similar, just narrow it down as appropriate and run that over a very wide time range or all time to determine if your events are going in with invalid timestamps or not....

0 Karma
Highlighted

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

Path Finder

Returns a count of zero, other two fields are blank, so I am guessing that it is not even being indexed?

0 Karma
Highlighted

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

Influencer

Can you run | tstats count where index=yourindexname by host and see if your source host is listed?
Also, check | metadata type=hosts to see if your firewall host is listed at all.
Finally, do you get any results if you run a 30-second windowed real-time search using index=yourindex host=yourmissingfirewall

0 Karma
Highlighted

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

Path Finder

I see it in the first two, but not the last. There seems to be a disparity though between last time and recent time..... Much more so than anything else.

0 Karma
Highlighted

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

Influencer

OK, so the tstats is showing events from that host in the given index, but when you are searching for data from that host, you are not coming up with anything?
I still suspect something is going on with the timestamps/timezone from that host. Can you compare the system settings for the 3 firewall devices and rule out that they differ in timezone settings?

0 Karma
Highlighted

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

Path Finder

Already did, timezone is the same across all 3. Time is the same as they are all synced using NTP. Funny, I found my events! Splunk is dating them as being 2 years prior! But I am positive the date on the firewalls are all the same.

0 Karma