How to change_time to indexing_time?

[Shaw]

Hi guys!

I need a help with a time problem. So  my structure is the following: i have many agent installed  on Windows machine that collects some data , i have a heavy forwarder thant handles the universals and forward data to an enterprise instances. 

My issue is that one of the server where the universal is installed has a time different from the other machine and from the heavy forwarder, in particular -1h .  So when i use search and alerts in real_time or in 5/10 minutes range i miss all the events related to that machine.

I would like all events to take as _time the system time of the enterprise instances (index_time)  or at least the heavy forwader system time. 

I tried to change the props.conf and inserting date_config = current at every level but nothing change.

It's ok also to have a custom configuration that add + 1h to that specif host  as long as the _time field is alligned with other machine.

Some assumption: All the machine are in the same country, the particular machine has different clock setting and can't be changed. The event that generates the universal contain always a timestamp of the specif machine but we don't want it as _time. 

[PickleRick]

Ok. Several things get combined here.

1. Don't use realtime searches. They are limited, they overuse resources.

2. Searching for a "near real-time" alerts is always prone to misses due to various forms of data "clogging" - most often, but not exclusively, caused by network outages.

3. Generally speaking, the "current" time config should rather be used as a last resort when there is no timestamp within the event (or when there is a timestamp but is completely unrelated to the event itself - like start time of a download in a download end event).

4. Completely apart from Splunk itself, it's a good practice to have your time configured properly and synchronized across your environment.

5. In a properly configured environment (look p.4 above) you should ingest events using provided timestamps and proper timezone configuration if the event itself doesn't contain TZ info.

As a side-note - Splunk configs are pretty case-sensitive, so if you configured "date_config=current", it won't work. Firstly, there's no option called "date_config", secondly, you're using lowercase.

[Shaw]

Hi thanks for the reply. 
I dont use real_time search it was an example to make clear the problem. Usally mny seach looks 5min or 10 min back for raise alarm e for visualizations.

I can't change the clock on my enviroment because it is has some implication on operations of that specific machine.

I used the configuration as explained in splunk documents so  DATETIME_CONFIG=CURRENT (sorry for the lower case) but seems not to work.

[PickleRick]

OK. Just wanted to let you know how it really _should_ be done because otherwise it can cause many strange issues (like the one you're having).

But back to the point.

What you need to do is to set the DATETIME_CONFIG=CURRENT in the appropriate place.

But which place is it?

Well, the first question is - on which component you need this to be set. I assume you're talking about "normal" windows eventlog/perflog data and internal splunk logs. The data is parsed on the first "heavy" (based on a full splunk enterprise installation package, not a UF) component in event's path. So if your setup is UF->HF->indexer(s), you need to set configure it on the HF.

And now the tricky part.

If you're collecting logs from the ForwardedEvents eventlog, I'm afraid I don't have good news - since the logs from ForwardedEvents are collected with the same host value for all hosts so you have no way of isolating a single host. Unless you do some magic like defining a metadata field in the UF and doing an ingest-time eval overwriting _time but that's a way more advanced topic than we're talking here.

So sticking to the basic option - your event by default has its raw data contents, the destination index, and three metadata fields - sourcetype, source, host. The sourcetype and source will (at least in any normal situation should) be the same for all your hosts. So the only field that can differentiate this "problematic" host from other ones is the host field.

So you should insert a stanza into the local/props.conf (either in etc/system or - preferably - in your custom app) with contents:

[host::my_host]
DATETIME_CONFIG=CURRENT

Of course substitute "my_host" with the value appropriate for your UF name (but watch out - the host field value can be overwritten in settings for any input on the UF - see my remark about the ForwardedEvents - all inputs reading from wineventlog://ForwardedEvents typically set host=WinEventLogForwardHost - that's why you can't tell one forwarding host from another). Typically you can see the host value "raw" in the internal events your UF sends to _internal index.

Final warning - since the "current" time configuration is being applied at the point of parsing, if you have any delay between UF and HF (for example due to network outage or HF restart), your data will have a wrong timestamp since the source-given time will be ignored and the HF-given timestamp will be applied at the time the data reaches HF's input.