Good afternoon,
We have 3 firewalls that are sending their syslogs to a udp port. 2 are showing events, one is not. It's like the events aren't being indexed. I tried sending the 1 firewall we aren't seeing data for to a different port mapped to the same source type to no avail. I do see splunkd listening on that port, and tcpdump shows data from that ip coming in on that port. What further steps may I take to diagnose this?
Thank you in advance!!!
Going into inputs.conf for search app and adding no_appending_timestamp = true for the stanza of the input fixed this issue.
Going into inputs.conf for search app and adding no_appending_timestamp = true for the stanza of the input fixed this issue.
Is the missing firewall sender coming from a different network, with spoofing of ip ?
It so, check this article : http://answers.splunk.com/answers/12876/splunk-running-on-my-linux-server-is-only-showing-me-events-...
Check splunkd.log on your indexers for any messages related to Timestamp parsing (Look for DateParserVerbose component). Is it possible that timestamps from the suspect firewall come in with a time in the future?
Also check within your servers metrics.log to see if that source is ever mentioned, the metrics log will mention the 10 busiest sources processed by this particular server...
Thank you both for your input! The source is mentioned in metrics.log. ssievert, I see no mention of timestamp parsing 😞
Can you run | tstats count where index=yourindexname by host
and see if your source host is listed?
Also, check | metadata type=hosts
to see if your firewall host is listed at all.
Finally, do you get any results if you run a 30-second windowed real-time search using index=yourindex host=yourmissingfirewall
I see it in the first two, but not the last. There seems to be a disparity though between last time and recent time..... Much more so than anything else.
OK, so the tstats is showing events from that host in the given index, but when you are searching for data from that host, you are not coming up with anything?
I still suspect something is going on with the timestamps/timezone from that host. Can you compare the system settings for the 3 firewall devices and rule out that they differ in timezone settings?
Already did, timezone is the same across all 3. Time is the same as they are all synced using NTP. Funny, I found my events! Splunk is dating them as being 2 years prior! But I am positive the date on the firewalls are all the same.
To add to that, the log entries themselves show the correct date and time.
That sounds like either automatic timestamp recognition is failing or your props.conf settings for the sourcetype are using a TIME_FORMAT that is not quite what you need.
Still stuck on this. Didn't see anything in props.conf that jumped out at me as being wrong. The 3 firewalls are the same make and model, running similar firmware. I do not know why the third would be timestamped incorrectly...
Can you post example log entries from the 3 firewalls ?
If they are using identical log format's and the same sourcetype you should not have an issue.
Also if you have customised the TIME_FORMAT OR TIME_PREFIX settings can you post them as well?
I am going to shy away from posting the actual log entries, but I can say they are from 3 different palo alto firewalls running the same exact version of PAN OS, time is synced with NTP, (checked time anyways) checked all syslog settings and they are identical.
Can you post what the time format looks like ? Also is there only 1 timestamp in the line?
Does the timestamp appear exactly the same between the 3 ?
The time format that splunk is stamping on it or the time from the actual events in the logs?
Finally fixed! Went into inputs.conf and added no_appending_timestamp = true and viola.
Thank you again everyone for your help and input!
The time appears in the log entries in several places
Oct 12 10:16:46
2017/10/12 10:16:46
Oct 12 10:16:46
2017/10/12 10:16:46
They format of the logs across the 3 firewalls are identical
Could you run:
| tstats count, max(_indextime) AS mostrecentindex, max(_time) AS mostrecent where index=<yourindex>, source=<yoursource>
| eval mostrecentindex=strftime(mostrecentindex, "%+"), mostrecent = strftime(mostrecent, "%+")
The where clause can have sourcetype/source/index or similar, just narrow it down as appropriate and run that over a very wide time range or all time to determine if your events are going in with invalid timestamps or not....
Returns a count of zero, other two fields are blank, so I am guessing that it is not even being indexed?