Event Time does not match

[phamanh1652]

We are using SC4S to collect local logs from FortiAnalyzer. We've noticed a error: the timestamp within the log file does not match the event time in Splunk.

This delay is causing a issue: when logs are first sent from FortiAnalyzer, they are not immediately searchable in Splunk. Instead, they only become searchable 7 hours later.

This problem appears to be isolated to the FortiAnalyzer local logs. All other log sources collected via SC4S are working correctly, even the log forwarded to FortiAnalyzer then to Splunk, with their log timestamps and event times matching perfectly.

How can we resolve this issue?

[phamanh1652]

Hello,

After configuring the SC4S server to use the same NTP server as the network devices, the timezone is now working correctly.

Thank you for all your help.

[PickleRick]

That means that you have/had another problem. NTP on its own doesn't care about time zones.

[PickleRick]

1. If the events are wrongly assigned timestamp they _are_ searchable but the default search range ends at "now" so those events do not fall into this range. Try searching with "latest=+12h" to see if the events are "properly" indexed into the future.

2. It seems like a timezone issue. What timezone your source is in? What timezone your SC4S runs in? What timezone your Splunk indexers (or HF if you're sending to HF) run in?

[isoutamo]

As @PickleRick said this is timezone issue.

Are all those logs wrongly timed or only some? I mean that if your SC4S is in one TZ and you are collecting syslogs from several different locations.  Also are your splunk servers and SC4S in same TZ?

There are at least two common syslog protocols:

• RFC3164 aka BSD syslog 
• RFC5424 

The newer (RFC5424) contains TZ information on every event, but old one has only date and time, but not TZ information. 

Check what those sources are used and if possible use RFC5424 version. If you cannot use that then you must add TZ information to those on SC4S or Splunk HEC side. Here is one instructions for it https://splunk.my.site.com/customer/s/article/Splunk-Connect-for-Syslog-Events

[PickleRick]

The event as shown (and as I remember Forti products) doesn't actually conform to either RFC - it's not strictly a syslog message. It's just "something" sent over the network. So unless SC4S can parse out the timestamp in this specific format (which I doubt but I don't have much experience here), it's left for Splunk to do.

[isoutamo]

For some reason I'm not surprised this with Forti products 😞

[phamanh1652]

Hello,

In our environment, we have Splunk Cloud, on-premise infrastructure including SC4S, and FortiAnalyzer. All systems are set to the same GMT+7 time zone. The issue is specific to the local logs from FortiAnalyzer.

We have the following add-ons installed:

• Fortinet FortiGate Add-on for Splunk (version 1.6.9)

• Fortinet FortiGate App for Splunk (version 1.6.4)

The problem only affects a specific type of log from FortiAnalyzer:

• Logs from other FortiGates: These logs are forwarded to FortiAnalyzer and then to Splunk. They are working correctly, and the log time matches the Splunk event time.

• Local logs from FortiAnalyzer: This includes events like login, logout, and configuration changes on the FortiAnalyzer itself. For these logs, there is a 7-hour time difference between the log timestamp and the Splunk event time.

This time discrepancy causes a significant problem. For example, if we create an alert for a configuration change on FortiAnalyzer, it will be triggered 7 hours late, making real-time monitoring impossible (As shown in this picture, using the same SPL query, searching by Splunk's event time returns results, while searching by the actual timestamp in the logs returns nothing.)

[PickleRick]

OK. Let me get this straight.

You have a single stream of events you're receving on your SC4S from the FortiAnalyzer and some of those events come directly from the FortiAnalyzer while other ones are forwarded by FortiAnalyzer from FortiGates? Is that correct?

I'm not aware that - without additional bending over backwards - SC4S can treat different events within single event stream differently.

Anyway, how is the timestamp rendered for both of those kinds of events? (in the original raw events)

[isoutamo]

Can you show sample of raw syslog events before those have sent to SC4S?
Can you also check the whole event from FortiAnalyzer (local logs) and logs from other FortiGates sent to FA and check if there was some other field where those times are set correctly? It's so long time ago when I have looked those that I cannot remember what all fields there are 😞 If I recall right there are many/some information several times in one event with little bit different format. Maybe there was another field which contains also TZ information?

[phamanh1652]

Hello,

Here is the raw local log of FortiAnalyzer, its timezone is also GMT+7

I checked the logs from FortiGate, which are forwarded to FortiAnalyzer and then to Splunk. When comparing these with the local logs of FortiAnalyzer, I noticed a key difference: the FortiGate logs contain timestamp, eventtime, and timezone, while the local FortiAnalyzer logs only show time.

[PickleRick]

So apparently you have two different event formats received from the same source, right?

One - and this one is properly parsed - contains both an absolute timestamp as well as timezone offset. The other one contains only time without a timezone definition so depending on your SC4S/Splunk configuration might simply treat the timestamp as GMT and apply the +7:00 offset to it.

I'm not an expert on SC4S but AFAIR it expects a single type of events for a single source so to "split" your processing path you need to do some additional conditional routing in the underlying syslog-ng configuration.

[phamanh1652]

Hello,

Thanks for your response. I've added SC4S_DEFAULT_TIMEZONE to the env_file, but the issue still persists. I came across a related article that suggests the problem might be due to a missing NTP server configuration. I’ll try setting that up and will follow up if it resolves the issue.

[PickleRick]

Nope. NTP client keeps sets the time to that received from the server but it doesn't manipulate timezone settings. It would help if you had the timestamp wrongly _set_ on a box, not wrongly interpreted.

Setting default timezone might also not help if SC4S doesn't understand the event format and can't make heads or tails of where the timestamp is.

[PrewinThomas]

@phamanh1652 

I don't think setting up NTP will fix this issue, as the fortianalyzer is logging time info only.  Is it possible for you to configure with RFC5424 logging(which should ideally have timezone info with the event).

Otherwise you’ll need a rewrite rule (with a filter for just the local fortianalyzer logs) that adds today’s date and your +07:00 timezone to the log’s time= value, which would add complete timestamp details to events before reaching Splunk.

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!