Splunk logs visible after 5hrs:30 mins in splunk UI- What could be wrong here?

[Yashprime07]

Splunk logs visible after 5hrs:30 mins in splunk UI for example , if I have to see the log of 13:00 to 14:00 , in UI I have to check for 18:00 to 19:00 .

Here splunk forwarder docker container works as a sidecar container alongside application container with same source volume mounted to both of the containers .

Can someone help what could be wrong here ??

[Yashprime07]

Thanks @PickleRick @isoutamo @ITWhisperer , the issue was that source type was not specified for this monitor, which breakdowns the events and adds the timezone IST to the events

[VatsalJagani]

@Yashprime07 - Is your question resolved? If so, kindly accept the answer by clicking on "Accept as Solution" button below the helpful answer.

Splunk Community Moderator,

Vatsal

[PickleRick]

Somewhere in this setup you have a misconfigured timezone.

First thing to do would be to check what timestamp is really produced by your source. See what timestamp is in the raw event - is it your local timestamp or maybe it's UTC? Does it include timezone information?

[Yashprime07]

Could this help ? @PickleRick @ITWhisperer 

[PickleRick]

OK. Apart from one more thing which might be interfering here - what is your user's timezone configured in splunk's webui in your account's preferences?

Anyway, since your sourcetype shows as "auth-too_small" it means that your logs onboarding is not properly configured.

So first things first:

1) Make sure what timestamps your data source produces (in your case - the app). I see "13:09" but what timezone is it in? UTC? Your local TZ? Unfortunately, the timestamp does not - as far as I can see - provide any notion of timezone which is not the best practice. Usually for clarity you'd want to either emit a timestamp containing TZ declaration or at least make sure that data is consistently reported in UTC (you could use local timezone but it might lead to problems with daylight saving if your TZ uses it)

2) Configure the forwarder and receiving indexer accordingly for the sourcetype - configure time recognition parameters - prefix, format, timezone if needed.

[Yashprime07]

Also, 
         Checked account preference for timezone, it's this  -

(GMT+05:30) Chennai, Kolkata, Mumbai, New Delhi

[Yashprime07]

1) It's sending in IST 
2) But I could see _time being 5hrs:30mins greater than the event timestamp, will check these forwarder parameters for that source. But currently it looks like this -

[monitor:///var/log/auth/app]
index = auth-signedcall
_meta = stack::<stackname>

[isoutamo]

Hi

As @PickleRick said your sourcetype hasn't onboarded correctly. Actually you have just add log file for collection, but You haven't define anything else for it.

You should add sourcetype into your inputs.conf and then with it add needed definitions on your 1st full Splunk Enterprise instance from docker container to indexers. That could be a heavy forwarder or indexer. Ad base sourcetype definitions under [<your sourcetype>] and then e.g. TZ definitions under [host::<your hosts which are in IST TZ>]

Those should fix your issue. But as PickleRick said, there could be still TZ issue when summer time changed to normal time and vice versa. So please check it also after that time.

r. Ismo

[Yashprime07]

Also, 
         we added this part in the application code's Dockerfile  too -

ENV TZ=Asia/Calcutta
RUN ln -snf /usr/share/zoneinfo/${TZ} /etc/localtime && echo ${TZ} > /etc/timezone

[Yashprime07]

Inside the container I get this -

bash-4.2# date
Mon Jun 26 12:54:44 IST 2023

[ITWhisperer]

Could it be a timezone setting mismatch?