calls indexing 'in the future' on a HEC using a custom source type

[loganramirez]

So...I have a HEC receiving JSON for phone calls using a custom sourcetype which parses calls from a field called timestampStr which looks like this:

2024-01-19 19:17:04.60313329

And uses TIME_FORMAT in the sourcetype with %Y-%m-%d %H:%M:%S.%9N

And this sets _time to 2024-01-19 19:17:04.603 in the event.

Which SEEMS RIGHT.

However, if I then, as a user in Central time zone, ask for the calls 'in the last 15 minutes' (assuming I just made the call) it does not show up.  And, in fact, to find it, I have to use ALL_TIME (because the call seems to be 'in the future' based on my user, which feels dumb/weird to write because I know it's not technically true, but the best I can explain it).

Here is what I mean (example):

1. i placed a call into the network at 1:17PM CENTRAL TIME (which is 19:17 UTC)
2. the application sent a json message of that call
3. it came in as above
4. i then ran a search in Splunk looking for that call in the LAST 15 MINUTES and it did not find it
5. however, i immediately asked for ALL_TIME and it did.

I think my assumption is that if my USER SETTINGS are set to Central, it would 'correlate the calls' ok, but this appears not true or, rather, more likely, my understanding of what is going on is very poor.

So I, once again, return to the community looking for answers.

[PickleRick · Accepted solution]

Which endpoint are you pushing your events to? /raw or /event?

The /event endpoint, unless called with ?auto_extract_timestamp=true parameter skips timestamp recognition completely.

Anyway, if there is no timezone info contained within the timestamp itself, Splunk sets the timezone for parsing the timestamp according to those settings https://docs.splunk.com/Documentation/Splunk/latest/Data/Applytimezoneoffsetstotimestamps#Specify_ti...

If you're sending to /raw endpoint you need to make sure you have proper timezone info set either for your whole forwarder where the HEC input is defined or for the particular sourcetype. If you're sending to the /event endpoint and you're not parsing the timestamp from the data, you need to make sure the proper timestamp (this time formatted as unix-timestamp) is being sent with the event or - if there is no timestamp info sent with the event (not _in_ the event - _with_ it), make sure that time (including timezone) is properly configured on the receiving forwarder.

[dural_yyz]

Your timestamp raw feed does not have a time zone indicator.  In that case the indexer will assume local system time zone and apply that in most cases.  If our collection point (UF) is -4 and your indexing point (IDX) is -5 it will assume that the timestamp is also -5.

A best practice is to always define your time zone and never allow Splunk to automagically assume based on the datetime.xml auto extractions.

https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/Propsconf#Timestamp_extraction_configuratio...

[loganramirez]

But 'assuming local time' is ok if the server is GMT/UTC, right, then the user's timezone should adjust?

This exact same sourcetype and configuration works at another installation, so something must be configured differently.

Here is the example of it 'working' (attached) where the timestampStr is UTC, but the _time is showing as Central Standard Time, which is my user defined timezone.

[PickleRick]

No. "Assuming local time if the server is UTC" (I assume 😉 you're talking about the parsing server, not the source) is OK only if the source sends the data in UTC.

[loganramirez]

yes definitely the parsing server!  and this is what i found out, the server was NOT set to UTC.  had them change and i'm gtg, however, I don't see how I can explicitly set the TZ on indexing, as it is not part of the string being sent to us and I do not see where we can set the TZ in the source type.

as always appreciate the education everyone, thank you!

[PickleRick]

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf#Timestamp_extraction_configurati...

You have the TZ parameter which you can use to "bind" a predefined timezone to a particular sourcetype, source or host. But it's always best if the source specifies the timezone within the timestamp itself - it saves you some work and possibly much grief later.

[dural_yyz]

The absolute best practice is to always explicitly define your timestamp along with timezone.  You can get away with it most times if you don't but when it doesn't work always revert back to a proper definition.