topic Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port in Getting Data In

Why isnt't our firewall showing events? We're sending syslogs to a UDP port

jb1982 — Tue, 10 Oct 2017 16:14:51 GMT

Good afternoon,
We have 3 firewalls that are sending their syslogs to a udp port. 2 are showing events, one is not. It's like the events aren't being indexed. I tried sending the 1 firewall we aren't seeing data for to a different port mapped to the same source type to no avail. I do see splunkd listening on that port, and tcpdump shows data from that ip coming in on that port. What further steps may I take to diagnose this?

Thank you in advance!!!

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

s2_splunk — Tue, 10 Oct 2017 19:34:56 GMT

Check splunkd.log on your indexers for any messages related to Timestamp parsing (Look for DateParserVerbose component). Is it possible that timestamps from the suspect firewall come in with a time in the future?

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

gjanders — Tue, 10 Oct 2017 19:55:41 GMT

Also check within your servers metrics.log to see if that source is ever mentioned, the metrics log will mention the 10 busiest sources processed by this particular server...

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

jb1982 — Tue, 10 Oct 2017 20:01:30 GMT

Thank you both for your input! The source is mentioned in metrics.log. ssievert, I see no mention of timestamp parsing 😞

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

gjanders — Tue, 10 Oct 2017 20:06:51 GMT

Could you run:

| tstats count, max(_indextime) AS mostrecentindex, max(_time) AS mostrecent where index=<yourindex>, source=<yoursource>
| eval mostrecentindex=strftime(mostrecentindex, "%+"), mostrecent = strftime(mostrecent, "%+")

The where clause can have sourcetype/source/index or similar, just narrow it down as appropriate and run that over a very wide time range or all time to determine if your events are going in with invalid timestamps or not....

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

s2_splunk — Tue, 10 Oct 2017 20:08:58 GMT

Can you run | tstats count where index=yourindexname by host and see if your source host is listed?
Also, check | metadata type=hosts to see if your firewall host is listed at all.
Finally, do you get any results if you run a 30-second windowed real-time search using index=yourindex host=yourmissingfirewall

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

jb1982 — Tue, 10 Oct 2017 20:10:06 GMT

Returns a count of zero, other two fields are blank, so I am guessing that it is not even being indexed?

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

jb1982 — Tue, 10 Oct 2017 20:17:53 GMT

I see it in the first two, but not the last. There seems to be a disparity though between last time and recent time..... Much more so than anything else.

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

s2_splunk — Tue, 10 Oct 2017 20:25:59 GMT

OK, so the tstats is showing events from that host in the given index, but when you are searching for data from that host, you are not coming up with anything?
I still suspect something is going on with the timestamps/timezone from that host. Can you compare the system settings for the 3 firewall devices and rule out that they differ in timezone settings?

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

jb1982 — Tue, 10 Oct 2017 20:28:02 GMT

Already did, timezone is the same across all 3. Time is the same as they are all synced using NTP. Funny, I found my events! Splunk is dating them as being 2 years prior! But I am positive the date on the firewalls are all the same.

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

jb1982 — Tue, 10 Oct 2017 20:28:42 GMT

To add to that, the log entries themselves show the correct date and time.

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

s2_splunk — Tue, 10 Oct 2017 20:33:12 GMT

That sounds like either automatic timestamp recognition is failing or your props.conf settings for the sourcetype are using a TIME_FORMAT that is not quite what you need.

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

jb1982 — Thu, 12 Oct 2017 13:05:18 GMT

Still stuck on this. Didn't see anything in props.conf that jumped out at me as being wrong. The 3 firewalls are the same make and model, running similar firmware. I do not know why the third would be timestamped incorrectly...

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

gjanders — Tue, 29 Sep 2020 16:14:01 GMT

Can you post example log entries from the 3 firewalls ?

If they are using identical log format's and the same sourcetype you should not have an issue.
Also if you have customised the TIME_FORMAT OR TIME_PREFIX settings can you post them as well?

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

jb1982 — Thu, 12 Oct 2017 13:24:56 GMT

I am going to shy away from posting the actual log entries, but I can say they are from 3 different palo alto firewalls running the same exact version of PAN OS, time is synced with NTP, (checked time anyways) checked all syslog settings and they are identical.

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

gjanders — Thu, 12 Oct 2017 13:32:43 GMT

Can you post what the time format looks like ? Also is there only 1 timestamp in the line?
Does the timestamp appear exactly the same between the 3 ?

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

jb1982 — Thu, 12 Oct 2017 14:22:50 GMT

The time appears in the log entries in several places
Oct 12 10:16:46
2017/10/12 10:16:46
Oct 12 10:16:46
2017/10/12 10:16:46

They format of the logs across the 3 firewalls are identical

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

jb1982 — Thu, 12 Oct 2017 14:24:26 GMT

The time format that splunk is stamping on it or the time from the actual events in the logs?

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

yannK — Thu, 12 Oct 2017 21:11:09 GMT

Is the missing firewall sender coming from a different network, with spoofing of ip ?
It so, check this article : http://answers.splunk.com/answers/12876/splunk-running-on-my-linux-server-is-only-showing-me-events-from-my-local-subnet-what-is-going-on

Re: Why isnt't our firewall showing events? We're sending syslogs to a UDP port

jb1982 — Tue, 29 Sep 2020 16:27:37 GMT

Finally fixed! Went into inputs.conf and added no_appending_timestamp = true and viola.
Thank you again everyone for your help and input!