Why UTC events suddenly started displaying as PST instead of Eastern Time?

[att35]

Hi,

We have data from Change Auditor coming via HEC setup on a Heavy Forwarder. This HF instance was upgraded to Version 9.2.2. After that, I am seeing a difference in the way Splunk displays new events on SH. It is now converting UTC->PST.  I ran a search for previous week and for those events it is converting timestamp correctly, from UTC-> Eastern. 

I am a little confused since both searches are done from same search head against same set of indexers. If there was a TZ issue, woudn't Splunk have converted both incorrectly?  I also ran same searches on indexer with identical output. Recent events in PST whereas older events continue to show as EST.

Here are some examples

For previous week

Recent. Splunk shows a UTC->PST conversion instead.



I did test this manually via Add Data and Splunk is correctly formatting it to Eastern. How can I troubleshoot why recent events in search are showing PST conversion?

My current TZ setting on SH is still set to Eastern Time. Also confirmed that system time for HF, indexers and Search Heads is set to Eastern. 

Thanks 

[PickleRick]

What are your setting for that sourcetype/source/host? And how are you pushing the events (to which endpoint)?

[att35]

@PickleRick Updated the post with the settings in place on HF.

Data is being received at Heavy Forwader via HEC input. It then gets forwarded to indexers. 

[PickleRick]

Yes, I understand that it's pushed to HEC input on the HF. But to which API endpoint? Because there are at at least three endpoints for the HEC input

/services/collector/raw

/services/collector/event

/services/collector/mint

Additionally the /event endpoint can accept parameters changing the ingestion process.

So I repeat my question - to which endpoint is your data being sent?

[att35]

@PickleRick 

It is sending to services/collector/event

[PickleRick]

So your timestamp extraction definition is not used because unless &auto_extract_timestamp=true is added to the /event URI, that endpoint skips timestamp extraction completely and uses the "time" field from the event's envelope or (if there isn't one) a current timestamp from the receiving component (in your case - the HF).

[att35]

@PickleRick 

I changed the URL to use raw endpoint. This seems to have fixed the timestamp but Splunk is now breaking the events at the timestamp fields.

I have added  KV_MODE = json for this sourcetype on both HF and SH but that did not fix the line breaking.

[PickleRick]

KV_MODE has nothing to do with line breaking.

And I'd expect that you simply don't have properly set up line breaker and you have line merging enabled. Which results in Splunk splitting your input stream at each line and then merges the lines back (which is also very ineffective performancewise).

[dural_yyz]

Please provide your props.conf stanza for the specific sourcetype.  In my experience this is an indication where UTC is not explicitly set and the local HF timezone is being used which is not Eastern.  I'm not saying that is the case here for sure because perhaps you do have the TZ explicitly set.

The golden rule is never let Splunk automagically guess the time.  It's right almost always but when it's not it can mess with production data at the worst times.

[att35]

@dural_yyz 

I dont see any specific settings for this sourcetype under local props.conf. I added TIME_PREFIX and TZ values but that didnt change anything. 

This is on the source which is getting the data, i.e. Heavy Forwarder. Do I need to place any of these settings on indexer/SH as well?

[change:auditor]
category = Custom
pulldown_type = 1
TIME_PREFIX = timeDetected
TZ = UTC

System time zone on HF is set to EDT

[dural_yyz]

TIME_PREFIX is a regex match for what immediately precedes your timestamp.  There are extra quotes, spaces, and what appears to be json key value pair identifiers.  I would make the value more explicit and add a MAX_TIMESTAMP_LOOKAHEAD key once you establish a proper match above.

[att35]

Thanks @dural_yyz 

Will try that.