topic Re: Why UTC events suddenly started displaying as PST instead of Eastern Time? in Getting Data In

Why UTC events suddenly started displaying as PST instead of Eastern Time?

att35 — Fri, 11 Oct 2024 16:19:56 GMT

Hi,

We have data from Change Auditor coming via HEC setup on a Heavy Forwarder. This HF instance was upgraded to Version 9.2.2. After that, I am seeing a difference in the way Splunk displays new events on SH. It is now converting UTC->PST. I ran a search for previous week and for those events it is converting timestamp correctly, from UTC-> Eastern.

I am a little confused since both searches are done from same search head against same set of indexers. If there was a TZ issue, woudn't Splunk have converted both incorrectly? I also ran same searches on indexer with identical output. Recent events in PST whereas older events continue to show as EST.

Here are some examples

For previous week

Recent. Splunk shows a UTC->PST conversion instead.

I did test this manually via Add Data and Splunk is correctly formatting it to Eastern. How can I troubleshoot why recent events in search are showing PST conversion?

My current TZ setting on SH is still set to Eastern Time. Also confirmed that system time for HF, indexers and Search Heads is set to Eastern.

Thanks

Re: Why UTC events suddenly started displaying as PST instead of Eastern Time?

dural_yyz — Fri, 11 Oct 2024 17:52:19 GMT

Please provide your props.conf stanza for the specific sourcetype. In my experience this is an indication where UTC is not explicitly set and the local HF timezone is being used which is not Eastern. I'm not saying that is the case here for sure because perhaps you do have the TZ explicitly set.

The golden rule is never let Splunk automagically guess the time. It's right almost always but when it's not it can mess with production data at the worst times.

Re: Why UTC events suddenly started displaying as PST instead of Eastern Time?

PickleRick — Fri, 11 Oct 2024 17:58:01 GMT

What are your setting for that sourcetype/source/host? And how are you pushing the events (to which endpoint)?

Re: Why UTC events suddenly started displaying as PST instead of Eastern Time?

att35 — Fri, 11 Oct 2024 18:27:37 GMT

@dural_yyz

I dont see any specific settings for this sourcetype under local props.conf. I added TIME_PREFIX and TZ values but that didnt change anything.

This is on the source which is getting the data, i.e. Heavy Forwarder. Do I need to place any of these settings on indexer/SH as well?

[change:auditor] category = Custom pulldown_type = 1 TIME_PREFIX = timeDetected TZ = UTC

System time zone on HF is set to EDT

Re: Why UTC events suddenly started displaying as PST instead of Eastern Time?

att35 — Fri, 11 Oct 2024 18:31:34 GMT

@PickleRick Updated the post with the settings in place on HF.

Data is being received at Heavy Forwader via HEC input. It then gets forwarded to indexers.

Re: Why UTC events suddenly started displaying as PST instead of Eastern Time?

dural_yyz — Fri, 11 Oct 2024 19:37:35 GMT

TIME_PREFIX is a regex match for what immediately precedes your timestamp. There are extra quotes, spaces, and what appears to be json key value pair identifiers. I would make the value more explicit and add a MAX_TIMESTAMP_LOOKAHEAD key once you establish a proper match above.

Re: Why UTC events suddenly started displaying as PST instead of Eastern Time?

PickleRick — Fri, 11 Oct 2024 20:57:33 GMT

Yes, I understand that it's pushed to HEC input on the HF. But to which API endpoint? Because there are at at least three endpoints for the HEC input

/services/collector/raw

/services/collector/event

/services/collector/mint

Additionally the /event endpoint can accept parameters changing the ingestion process.

So I repeat my question - to which endpoint is your data being sent?

Re: Why UTC events suddenly started displaying as PST instead of Eastern Time?

att35 — Fri, 11 Oct 2024 22:31:46 GMT

Thanks @dural_yyz

Will try that.

Re: Why UTC events suddenly started displaying as PST instead of Eastern Time?

att35 — Fri, 11 Oct 2024 22:35:01 GMT

@PickleRick

It is sending to services/collector/event

Re: Why UTC events suddenly started displaying as PST instead of Eastern Time?

PickleRick — Fri, 11 Oct 2024 22:58:51 GMT

So your timestamp extraction definition is not used because unless &auto_extract_timestamp=true is added to the /event URI, that endpoint skips timestamp extraction completely and uses the "time" field from the event's envelope or (if there isn't one) a current timestamp from the receiving component (in your case - the HF).

Re: Why UTC events suddenly started displaying as PST instead of Eastern Time?

att35 — Thu, 24 Oct 2024 15:31:16 GMT

@PickleRick

I changed the URL to use raw endpoint. This seems to have fixed the timestamp but Splunk is now breaking the events at the timestamp fields.

I have added KV_MODE = json for this sourcetype on both HF and SH but that did not fix the line breaking.

Re: Why UTC events suddenly started displaying as PST instead of Eastern Time?

PickleRick — Thu, 24 Oct 2024 18:34:12 GMT

KV_MODE has nothing to do with line breaking.

And I'd expect that you simply don't have properly set up line breaker and you have line merging enabled. Which results in Splunk splitting your input stream at each line and then merges the lines back (which is also very ineffective performancewise).