How to configure props.conf to set the universal forwarder's server time as the event timestamp?

[arkadyz1]

I'm trying to solve the following problem: in our client's environment, the clocks on different servers can vary greatly. We can easily have a server which is 3 hours behind on its system clock. And it's not a timezone issue - the servers are all supposed to have the same time.

All servers have univeral forwarders sending events to some Splunk Enterprise instances. We want each event to have the time of its import by the universal forwarder as a timestamp. Will setting DATETIME_CONFIG=CURRENT on the forwarder and DATETIME_CONFIG=NONE on the indexer produce the desired result?

[hlarimer]

We actually alert every day on this very thing...when timestamps are incorrect. What I recommend doing is using a search that tells you when timestamps are off and it will allow you to keep up on this. I find existing or new systems with incorrect timestamps on a regular basis (new sites, system rebuilds, etc) but now that we keep up on them its much easier to manage. Here is the search I use for the alert:

| metadata type=hosts index=*  | eval diff=recentTime-lastTime | where diff > 1800 OR diff < -1800 | convert ctime(lastTime) AS last_log_date |  table host diff last_log_date 

This search looks for the difference between the recentTime and the lastTime which is the difference of when the log came in versus what it was timestamped. There will always be some difference based on lag so I only alert if its off by 1800 seconds or more. The negative 1800 is important too since it captures timestamps that are set in the future.

[arkadyz1 · Accepted solution]

OK, first of all to answer the questions about "why not synchronize the clocks" etc. - it is an environment which is out of our control, so instead of trying to fix the problem, we are trying to capture the symptom to be able to report it to our customer.

Second, those settings I suggested, applied to the whole sourcetype (you cannot apply more granularity and go by the source, for example), produced the desired result.

Thanks for all the replies!

[kristian_kolb]

Unless you are batching in your logs somehow to the UF, or have SEVERE file monitor lag on your UF's, the event timestamp (as written by some application) and the UF "read-timestamp" should be fairly close to each other. Within seconds - easily - for most normal situations.

The UF program does not maintain its own clock, it uses system time anyway. So if the UF is reading the logs in a timely manner, you would not gain anything.

It looks like you are trying to fix the symptom, rather than the problem, as woodcock suggests.

[woodcock]

No, you will have to use Heavy Forwarders and use DATETIME_CONFIG=CURRENT there and nothing else on the indexers. IMHO, this is a terrible idea and the effort ahould be spent on NTP.

[somesoni2]

Should the forwarders have same time set on clock as indexer?? If yes, then set the DATETIME_CONFIG=CURRENT on props.conf, under sourcetype/source/host stanza to set all event timestamp to current time on Indexer.