topic Re: Splunk logs visible after 5hrs:30 mins in splunk UI in Getting Data In

Splunk logs visible after 5hrs:30 mins in splunk UI- What could be wrong here?

Yashprime07 — Mon, 26 Jun 2023 15:42:42 GMT

Splunk logs visible after 5hrs:30 mins in splunk UI for example , if I have to see the log of 13:00 to 14:00 , in UI I have to check for 18:00 to 19:00 .

Here splunk forwarder docker container works as a sidecar container alongside application container with same source volume mounted to both of the containers .

Can someone help what could be wrong here ??

Re: Splunk logs visible after 5hrs:30 mins in splunk UI

ITWhisperer — Mon, 26 Jun 2023 05:45:50 GMT

Could it be a timezone setting mismatch?

Re: Splunk logs visible after 5hrs:30 mins in splunk UI

PickleRick — Mon, 26 Jun 2023 05:49:27 GMT

Somewhere in this setup you have a misconfigured timezone.

First thing to do would be to check what timestamp is really produced by your source. See what timestamp is in the raw event - is it your local timestamp or maybe it's UTC? Does it include timezone information?

Re: Splunk logs visible after 5hrs:30 mins in splunk UI

Yashprime07 — Mon, 26 Jun 2023 07:07:08 GMT

Could this help ? @PickleRick @ITWhisperer

Re: Splunk logs visible after 5hrs:30 mins in splunk UI

Yashprime07 — Mon, 26 Jun 2023 07:09:59 GMT

Also,
we added this part in the application code's Dockerfile too -

ENV TZ=Asia/Calcutta RUN ln -snf /usr/share/zoneinfo/${TZ} /etc/localtime && echo ${TZ} > /etc/timezone

Re: Splunk logs visible after 5hrs:30 mins in splunk UI

Yashprime07 — Mon, 26 Jun 2023 07:25:37 GMT

Inside the container I get this -

bash-4.2# date Mon Jun 26 12:54:44 IST 2023

Re: Splunk logs visible after 5hrs:30 mins in splunk UI

PickleRick — Mon, 26 Jun 2023 07:28:38 GMT

OK. Apart from one more thing which might be interfering here - what is your user's timezone configured in splunk's webui in your account's preferences?

Anyway, since your sourcetype shows as "auth-too_small" it means that your logs onboarding is not properly configured.

So first things first:

1) Make sure what timestamps your data source produces (in your case - the app). I see "13:09" but what timezone is it in? UTC? Your local TZ? Unfortunately, the timestamp does not - as far as I can see - provide any notion of timezone which is not the best practice. Usually for clarity you'd want to either emit a timestamp containing TZ declaration or at least make sure that data is consistently reported in UTC (you could use local timezone but it might lead to problems with daylight saving if your TZ uses it)

2) Configure the forwarder and receiving indexer accordingly for the sourcetype - configure time recognition parameters - prefix, format, timezone if needed.

Re: Splunk logs visible after 5hrs:30 mins in splunk UI

Yashprime07 — Mon, 26 Jun 2023 07:40:31 GMT

1) It's sending in IST
2) But I could see _time being 5hrs:30mins greater than the event timestamp, will check these forwarder parameters for that source. But currently it looks like this -

[monitor:///var/log/auth/app] index = auth-signedcall _meta = stack::<stackname>

Re: Splunk logs visible after 5hrs:30 mins in splunk UI

Yashprime07 — Mon, 26 Jun 2023 07:48:47 GMT

Also,
Checked account preference for timezone, it's this -

(GMT+05:30) Chennai, Kolkata, Mumbai, New Delhi

Re: Splunk logs visible after 5hrs:30 mins in splunk UI

isoutamo — Mon, 26 Jun 2023 07:50:14 GMT

As @PickleRick said your sourcetype hasn't onboarded correctly. Actually you have just add log file for collection, but You haven't define anything else for it.

You should add sourcetype into your inputs.conf and then with it add needed definitions on your 1st full Splunk Enterprise instance from docker container to indexers. That could be a heavy forwarder or indexer. Ad base sourcetype definitions under [<your sourcetype>] and then e.g. TZ definitions under [host::<your hosts which are in IST TZ>]

Those should fix your issue. But as PickleRick said, there could be still TZ issue when summer time changed to normal time and vice versa. So please check it also after that time.

r. Ismo

Re: Splunk logs visible after 5hrs:30 mins in splunk UI

Yashprime07 — Mon, 26 Jun 2023 14:09:36 GMT

Thanks @PickleRick @isoutamo @ITWhisperer , the issue was that source type was not specified for this monitor, which breakdowns the events and adds the timezone IST to the events

Re: Splunk logs visible after 5hrs:30 mins in splunk UI

VatsalJagani — Tue, 27 Jun 2023 05:44:52 GMT

@Yashprime07 - Is your question resolved? If so, kindly accept the answer by clicking on "Accept as Solution" button below the helpful answer.

Splunk Community Moderator,

Vatsal