Getting Data In

Why isnt't our firewall showing events? We're sending syslogs to a UDP port

jb1982
Path Finder

Good afternoon,
We have 3 firewalls that are sending their syslogs to a udp port. 2 are showing events, one is not. It's like the events aren't being indexed. I tried sending the 1 firewall we aren't seeing data for to a different port mapped to the same source type to no avail. I do see splunkd listening on that port, and tcpdump shows data from that ip coming in on that port. What further steps may I take to diagnose this?

Thank you in advance!!!

0 Karma
1 Solution

jb1982
Path Finder

Going into inputs.conf for search app and adding no_appending_timestamp = true for the stanza of the input fixed this issue.

View solution in original post

0 Karma

jb1982
Path Finder

Going into inputs.conf for search app and adding no_appending_timestamp = true for the stanza of the input fixed this issue.

0 Karma

yannK
Splunk Employee
Splunk Employee

Is the missing firewall sender coming from a different network, with spoofing of ip ?
It so, check this article : http://answers.splunk.com/answers/12876/splunk-running-on-my-linux-server-is-only-showing-me-events-...

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Check splunkd.log on your indexers for any messages related to Timestamp parsing (Look for DateParserVerbose component). Is it possible that timestamps from the suspect firewall come in with a time in the future?

gjanders
SplunkTrust
SplunkTrust

Also check within your servers metrics.log to see if that source is ever mentioned, the metrics log will mention the 10 busiest sources processed by this particular server...

0 Karma

jb1982
Path Finder

Thank you both for your input! The source is mentioned in metrics.log. ssievert, I see no mention of timestamp parsing 😞

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Can you run | tstats count where index=yourindexname by host and see if your source host is listed?
Also, check | metadata type=hosts to see if your firewall host is listed at all.
Finally, do you get any results if you run a 30-second windowed real-time search using index=yourindex host=yourmissingfirewall

0 Karma

jb1982
Path Finder

I see it in the first two, but not the last. There seems to be a disparity though between last time and recent time..... Much more so than anything else.

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

OK, so the tstats is showing events from that host in the given index, but when you are searching for data from that host, you are not coming up with anything?
I still suspect something is going on with the timestamps/timezone from that host. Can you compare the system settings for the 3 firewall devices and rule out that they differ in timezone settings?

0 Karma

jb1982
Path Finder

Already did, timezone is the same across all 3. Time is the same as they are all synced using NTP. Funny, I found my events! Splunk is dating them as being 2 years prior! But I am positive the date on the firewalls are all the same.

0 Karma

jb1982
Path Finder

To add to that, the log entries themselves show the correct date and time.

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

That sounds like either automatic timestamp recognition is failing or your props.conf settings for the sourcetype are using a TIME_FORMAT that is not quite what you need.

0 Karma

jb1982
Path Finder

Still stuck on this. Didn't see anything in props.conf that jumped out at me as being wrong. The 3 firewalls are the same make and model, running similar firmware. I do not know why the third would be timestamped incorrectly...

0 Karma

gjanders
SplunkTrust
SplunkTrust

Can you post example log entries from the 3 firewalls ?

If they are using identical log format's and the same sourcetype you should not have an issue.
Also if you have customised the TIME_FORMAT OR TIME_PREFIX settings can you post them as well?

0 Karma

jb1982
Path Finder

I am going to shy away from posting the actual log entries, but I can say they are from 3 different palo alto firewalls running the same exact version of PAN OS, time is synced with NTP, (checked time anyways) checked all syslog settings and they are identical.

0 Karma

gjanders
SplunkTrust
SplunkTrust

Can you post what the time format looks like ? Also is there only 1 timestamp in the line?
Does the timestamp appear exactly the same between the 3 ?

0 Karma

jb1982
Path Finder

The time format that splunk is stamping on it or the time from the actual events in the logs?

0 Karma

jb1982
Path Finder

Finally fixed! Went into inputs.conf and added no_appending_timestamp = true and viola.
Thank you again everyone for your help and input!

0 Karma

jb1982
Path Finder

The time appears in the log entries in several places
Oct 12 10:16:46
2017/10/12 10:16:46
Oct 12 10:16:46
2017/10/12 10:16:46

They format of the logs across the 3 firewalls are identical

0 Karma

gjanders
SplunkTrust
SplunkTrust

Could you run:

| tstats count, max(_indextime) AS mostrecentindex, max(_time) AS mostrecent where index=<yourindex>, source=<yoursource>
| eval mostrecentindex=strftime(mostrecentindex, "%+"), mostrecent = strftime(mostrecent, "%+")

The where clause can have sourcetype/source/index or similar, just narrow it down as appropriate and run that over a very wide time range or all time to determine if your events are going in with invalid timestamps or not....

0 Karma

jb1982
Path Finder

Returns a count of zero, other two fields are blank, so I am guessing that it is not even being indexed?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...