Getting wrong time for event?

[olivera]

I have an add on for unix and linux downloaded on my monitored servers and the data is sent to my indexers.

In the Unix:Service sourcetype the time that is wrriten is 3.5 hours delayed, meanwhile the time that is wrriten in the event itself is the correct time.

Can someone help please and know how to fix it???

[matt8679]

Hi,

I would check your user Preferences timezone.

Click your name in the top right > Preferences > Default System Timezone.

If you have it set to a timezone, it will convert the time for you.

Matt

[PickleRick]

Hard to say without seeing sample of your data and your configuration but there are several possible causes.

Most probably the timestamp is not being parsed from the event at all but is assumed to be the time of ingestion of data. But that's just a blind guess.

[olivera]

I assume it's caused because of the IST in the event but I don't know how to change it.

Can you help?

[PickleRick]

It seems pretty ok.

The event contains timestamp "9:50:51 IST". It is equivalent to "4:20:51 GMT" or "6:20:51 +2:00".

If the timestamp is wrong, it means your source is reporting wrong time and it's up to the source solution's admin to set the proper timestamp.

[gcusello]

Hi @olivera,

share you sample events as text not with a screenshot , so it's possible to use them.

is there in your events another timestamp?

anyway I see that you have events from Central India, probably you setted a different timezone or a wrong timestamp definition, please share the Unix:Service stanza of your props.conf.

It should be (with also other settings):

[Unix:Service]
TIME_PREFIX = ^
TIME_FORMAT = %a %b %d %H:%M:%S %Z %Y
MAX_TIMESTAMP_LOOKAHEAD = 30

 Ciao.

Giuseppe

[olivera]

I added the configuration and it did not change anything. Should it be in /opt/splunk/etc/deployment-apps/Splunk_TA_nix-default?

[gcusello]

Hi @olivera,

this configuration must be on the forwarders used to take data and on intermediate Heavy Forwarders (if present) or on Indexers, not on the deployment-apps folder, but in the $SPLUNK_HOME/etc/apps folder.

How do you manage your Forwarders? are you using a Deployment Server?

have you an Indexers Cluster?

Have you intermediate heavy Forwarders?

Based on these information it's possible to define how to deploy this configuration, anyway, the deployment-apps folder is only used by the Deployment Server to deploy apps to forwarders, so you have to put the configuration in this folder only if you're working on the DS and you want to deploy apps to Forwarders.

Ciao.

Giuseppe

[olivera]

I just found out that my time zone should be GMT +2 and it is written IST instead. Is there a way to change it in the event so the right time (which appears already in the event itself but not in the timestamp) will be parsed with this timezone?

Thank you in advance

[PickleRick]

Wait a minute. Your event has an IST timestamp. So it should be processed as such. If your source emits events with wrong timezone specification, it's something that should be fixed on the source. If it's telling you that something happened on midnight IST, why would you interpret it as midnight PDT, CET or any other timezone? The source is telling you that it's IST so it's interpreted as IST. Fix your source! (the problem with timezone setting on the source can be causing problems elsewhere as well, not only in your splunk).

[olivera]

Hello @PickleRick 

Actually my source is correct. I live in Israel and IST stands for Israel standard time but for some reason splunk interprets it as India standard time (also IST), which I believe creates the problem in the parsing of the raw text of the event.

I tried to configure TZ ALIAS in the props.conf but after many syntax tries it looks like it doesn't want to be applied. The props file is in an app which is on my UF's. I restarted the UF's and it still did not work.

What should I do to override the IST in the event?

pictures of props.conf:

[PickleRick]

You're right, that's probably the cause. I must say I'm a bit surprised - didn't know about such ambiguity in tz markings.

Anyway, your idea of TZ_ALIAS is good (don't set TZ for a fixed value - it will break your time in case of daylight saving). But it has to be placed where the timestamp is parsed from the event - indexers or HF if you're parsing on HF. UFs don't parse events (mostly).

[olivera]

@PickleRick 

You made things clear for me.

I have indexers and intermediate forwarders, but I configured them long ago. How can I know where the parsing of my data is? In addition, according to my knowledge, my app is only deployed to my UFs, so in what directory should I put my props.conf file? (currently in the UFs it's in ./app_name/local/props.conf)

In addition, I didn't understand what you meant by the TZ and daylight saving. Can you please explain?

Thank you

[PickleRick]

OK. I got a bit too far ahead of myself 🙂

The TZ setting is only applied if there is no timezone specified within the timestamp parsed from the event so in your case this setting - even though it is set - will not apply because you have your "IST" in your time string.

Time is generally being parsed on the first "heavy" (based on a full installation, not the universal forwarder binary) component in event's path. So if you're sending your events straight from UF to indexers, the events are parsed on indexers (including timestamp recognition and parsing). If you're sending them from UF to HF(s) which are sending to indexers, parsing is done on HFs. And on those parsing components (in the first case - indexers, in second - on HFs) you need the TZ_ALIAS config.

So if you deploy your config only to UFs, it won't work since UFs don't parse events. They just read/receive them (depending on the type of input) and forward them to indexers/HFs.

[gcusello]

Hi @olivera,

To know where you are aparsing your data you can see the outputs.conf that you deployed to the UFs.

Anyway, if you haven't it, I hint to design a chart of your distributed architetcture, otherwise it's very difficoult to correctly deploy Add-Ons to the Forwarders.

About the directory,the best approach is to manually copy or (better) deploy by Deploym,ent server the same Add-On also to the Heavy Forwarder.

At least, about TZ and daylight saving, I suppose that he mens maintaining the original configuration.

Ciao.

Giuseppe

[olivera]

Hello again @gcusello 

I put it in the deployment apps because then I deploy it to from the DS to my universal forwarders. I meant that after the deploy of the new configuration the time of the source type still did not change. Do you have another solution? I'm a bit clueless. It's weird that only one source type's time is wrong. 

Thank you for your answers

[gcusello]

Hi @olivera,

did you deployed the eabove configurations to all Universal Forwarders, Indexers and (if present) to intermediate Heavy Forwarders?

Ciao.

Giuseppe

P.S.: have an Happy New Year!

[olivera]

Hello again @gcusello 

In what directory should I put the props.conf file in the HF and Indexers? It's confusing because it is now located in the Add-On directory and I don't deploy this Add-On to those servers.

Wish you luck in 2023!

[gcusello]

Hi @olivera,

maybe in the event there are more timestamps and Splunk takes the wrong one or maybe there's a timezone, can you share some sampe of your events?

Ciao.

Giuseppe

[olivera]

I assume it's caused because of the IST in the event but I don't know how to change it. Can you help please?