topic Re: Getting wrong time for event in Getting Data In

Getting wrong time for event?

olivera — Thu, 29 Dec 2022 04:10:49 GMT

I have an add on for unix and linux downloaded on my monitored servers and the data is sent to my indexers.

In the Unix:Service sourcetype the time that is wrriten is 3.5 hours delayed, meanwhile the time that is wrriten in the event itself is the correct time.

Can someone help please and know how to fix it???

Re: Getting wrong time for event

gcusello — Tue, 27 Dec 2022 07:47:49 GMT

Hi @olivera,

maybe in the event there are more timestamps and Splunk takes the wrong one or maybe there's a timezone, can you share some sampe of your events?

Ciao.

Giuseppe

Re: Getting wrong time for event

PickleRick — Tue, 27 Dec 2022 09:02:14 GMT

Hard to say without seeing sample of your data and your configuration but there are several possible causes.

Most probably the timestamp is not being parsed from the event at all but is assumed to be the time of ingestion of data. But that's just a blind guess.

Re: Getting wrong time for event

olivera — Wed, 28 Dec 2022 08:13:52 GMT

I assume it's caused because of the IST in the event but I don't know how to change it.

Can you help?

Re: Getting wrong time for event

olivera — Wed, 28 Dec 2022 08:18:23 GMT

I assume it's caused because of the IST in the event but I don't know how to change it. Can you help please?

Re: Getting wrong time for event

gcusello — Wed, 28 Dec 2022 08:37:49 GMT

Hi @olivera,

share you sample events as text not with a screenshot , so it's possible to use them.

is there in your events another timestamp?

anyway I see that you have events from Central India, probably you setted a different timezone or a wrong timestamp definition, please share the Unix:Service stanza of your props.conf.

It should be (with also other settings):

[Unix:Service] TIME_PREFIX = ^ TIME_FORMAT = %a %b %d %H:%M:%S %Z %Y MAX_TIMESTAMP_LOOKAHEAD = 30

Ciao.

Giuseppe

Re: Getting wrong time for event

PickleRick — Wed, 28 Dec 2022 08:51:55 GMT

It seems pretty ok.

The event contains timestamp "9:50:51 IST". It is equivalent to "4:20:51 GMT" or "6:20:51 +2:00".

If the timestamp is wrong, it means your source is reporting wrong time and it's up to the source solution's admin to set the proper timestamp.

Re: Getting wrong time for event

olivera — Wed, 28 Dec 2022 09:34:26 GMT

I added the configuration and it did not change anything. Should it be in /opt/splunk/etc/deployment-apps/Splunk_TA_nix-default?

Re: Getting wrong time for event

gcusello — Wed, 28 Dec 2022 09:44:07 GMT

Hi @olivera,

this configuration must be on the forwarders used to take data and on intermediate Heavy Forwarders (if present) or on Indexers, not on the deployment-apps folder, but in the $SPLUNK_HOME/etc/apps folder.

How do you manage your Forwarders? are you using a Deployment Server?

have you an Indexers Cluster?

Have you intermediate heavy Forwarders?

Based on these information it's possible to define how to deploy this configuration, anyway, the deployment-apps folder is only used by the Deployment Server to deploy apps to forwarders, so you have to put the configuration in this folder only if you're working on the DS and you want to deploy apps to Forwarders.

Ciao.

Giuseppe

Re: Getting wrong time for event

matt8679 — Wed, 28 Dec 2022 17:14:24 GMT

Hi,

I would check your user Preferences timezone.

Click your name in the top right > Preferences > Default System Timezone.

If you have it set to a timezone, it will convert the time for you.

Matt

Re: Getting wrong time for event

olivera — Sat, 31 Dec 2022 17:18:19 GMT

Hello again @gcusello

I put it in the deployment apps because then I deploy it to from the DS to my universal forwarders. I meant that after the deploy of the new configuration the time of the source type still did not change. Do you have another solution? I'm a bit clueless. It's weird that only one source type's time is wrong.

Thank you for your answers

Re: Getting wrong time for event

olivera — Sat, 31 Dec 2022 17:37:54 GMT

I just found out that my time zone should be GMT +2 and it is written IST instead. Is there a way to change it in the event so the right time (which appears already in the event itself but not in the timestamp) will be parsed with this timezone?

Thank you in advance

Re: Getting wrong time for event

PickleRick — Sat, 31 Dec 2022 20:22:22 GMT

Wait a minute. Your event has an IST timestamp. So it should be processed as such. If your source emits events with wrong timezone specification, it's something that should be fixed on the source. If it's telling you that something happened on midnight IST, why would you interpret it as midnight PDT, CET or any other timezone? The source is telling you that it's IST so it's interpreted as IST. Fix your source! (the problem with timezone setting on the source can be causing problems elsewhere as well, not only in your splunk).

Re: Getting wrong time for event

olivera — Sat, 31 Dec 2022 20:52:59 GMT

Hello @PickleRick

Actually my source is correct. I live in Israel and IST stands for Israel standard time but for some reason splunk interprets it as India standard time (also IST), which I believe creates the problem in the parsing of the raw text of the event.

I tried to configure TZ ALIAS in the props.conf but after many syntax tries it looks like it doesn't want to be applied. The props file is in an app which is on my UF's. I restarted the UF's and it still did not work.

What should I do to override the IST in the event?

pictures of props.conf:

Re: Getting wrong time for event

PickleRick — Sat, 31 Dec 2022 23:59:53 GMT

You're right, that's probably the cause. I must say I'm a bit surprised - didn't know about such ambiguity in tz markings.

Anyway, your idea of TZ_ALIAS is good (don't set TZ for a fixed value - it will break your time in case of daylight saving). But it has to be placed where the timestamp is parsed from the event - indexers or HF if you're parsing on HF. UFs don't parse events (mostly).

Re: Getting wrong time for event

gcusello — Sun, 01 Jan 2023 07:02:04 GMT

Hi @olivera,

did you deployed the eabove configurations to all Universal Forwarders, Indexers and (if present) to intermediate Heavy Forwarders?

Ciao.

Giuseppe

P.S.: have an Happy New Year!

Re: Getting wrong time for event

olivera — Sun, 01 Jan 2023 22:03:29 GMT

@PickleRick

You made things clear for me.

I have indexers and intermediate forwarders, but I configured them long ago. How can I know where the parsing of my data is? In addition, according to my knowledge, my app is only deployed to my UFs, so in what directory should I put my props.conf file? (currently in the UFs it's in ./app_name/local/props.conf)

In addition, I didn't understand what you meant by the TZ and daylight saving. Can you please explain?

Thank you

Re: Getting wrong time for event

olivera — Sun, 01 Jan 2023 22:09:45 GMT

Hello again @gcusello

In what directory should I put the props.conf file in the HF and Indexers? It's confusing because it is now located in the Add-On directory and I don't deploy this Add-On to those servers.

Wish you luck in 2023!

Re: Getting wrong time for event

gcusello — Mon, 02 Jan 2023 08:08:58 GMT

Hi @olivera,

To know where you are aparsing your data you can see the outputs.conf that you deployed to the UFs.

Anyway, if you haven't it, I hint to design a chart of your distributed architetcture, otherwise it's very difficoult to correctly deploy Add-Ons to the Forwarders.

About the directory,the best approach is to manually copy or (better) deploy by Deploym,ent server the same Add-On also to the Heavy Forwarder.

At least, about TZ and daylight saving, I suppose that he mens maintaining the original configuration.

Ciao.

Giuseppe

Re: Getting wrong time for event

PickleRick — Mon, 02 Jan 2023 09:06:48 GMT

OK. I got a bit too far ahead of myself 🙂

The TZ setting is only applied if there is no timezone specified within the timestamp parsed from the event so in your case this setting - even though it is set - will not apply because you have your "IST" in your time string.

Time is generally being parsed on the first "heavy" (based on a full installation, not the universal forwarder binary) component in event's path. So if you're sending your events straight from UF to indexers, the events are parsed on indexers (including timestamp recognition and parsing). If you're sending them from UF to HF(s) which are sending to indexers, parsing is done on HFs. And on those parsing components (in the first case - indexers, in second - on HFs) you need the TZ_ALIAS config.

So if you deploy your config only to UFs, it won't work since UFs don't parse events. They just read/receive them (depending on the type of input) and forward them to indexers/HFs.