About gsonal03

gsonal03 · ‎04-21-2019

For changing at application level, I will need to evaluate if log4j can accept and interpret "TZ" in its configuration, else we may need to go the "props.conf" route suggested below.

gsonal03 · ‎04-21-2019

Thanks for the clarification about how timezone issue I am seen is by design. I will check with the team who has control of props.conf and enquire if anyone knows how to engage PS for our account.

gsonal03 · ‎04-19-2019

Thanks for correcting. I asked this "question" to understand if my interpretation that a negative time delay is coming due to server being in UTC-X hours or there could be some other potential problem, that I need to check on splunk configuration end? Are you able to provide any thoughts around that?

gsonal03 · ‎04-19-2019

After reviewing splund.log, metrics.log in several attempts and adding check on storage etc. on splunk servers, we have received one pattern. The only logs getting delayed by forwarder are the one's generated by multiple threads in an application. We have several single threaded applications that generate huge logs, but they don't get delayed during indexing. While the one app running parallel jobs in multiple threads and generating relatively less logs, still gets delayed in indexing. Is there any throttle limit that can be adjusted for allowing more logs to get forwarded to indexers per second?

gsonal03 · ‎04-19-2019

This ticket is to understand how splunk indextime calculations occur.

gsonal03 · ‎04-19-2019

We have a server running in Japan timezone. Recently when we did not find logs during a live testing. Next day we ran the query to calculate delta between indextime and event time --- eval delta=_indextime-_time It is showing even a negative value. Please refer to attached screenshot. Since that is not possible in real time that an index gets created even before event occurs, is there an issue because the server is behind UTC? 2019-04-18 13:49:20.992 || 2019-04-18 14:08:13 || 1132.008 2019-04-18 12:50:37.005 || 2019-04-18 14:08:13 || 4655.995 2019-04-18 13:49:21.046 || 2019-04-18 13:49:26 || 4.954 2019-04-18 13:49:21.038 || 2019-04-18 13:49:23 || 1.962 2019-04-18 21:53:45.843 || 2019-04-18 12:53:51 || **** -32394.843 **** 2019-04-18 12:52:04.591 || 2019-04-18 12:52:05 || 0.409

gsonal03 · ‎03-16-2019

Thanks for the explanation. I am not blaming splunk for anything, just trying to understand so it can utilized in correct manner. With the explanation you are giving, it seems the source log file is logging in EST, that would mean the server which I assumed was in GMT is in fact in EST location. So, I need to change my account settings to EST then, to get consistent logs. I will try this and see if it helps in finding old logs in appropriate date time range.

gsonal03 · ‎03-16-2019

I am trying to debug issues related to delay in splunk forwarding or indexing in a separate splunk query "https://answers.splunk.com/answers/730136/why-are-our-splunk-indexes-not-showing-all-log-ent.html. But I would like to understand how the display of raw logs are governed, so opening a new ticket. Attached below is a mockup of how I see logs in raw format and account settings. I have my account settings configured to GMT timezone. When I search any logs in raw format, I see each log entry beginning with EST timestamp. When I expand it, I see _time field showing time in GMT format. How and where can I change the settings for the log entry so that it remains consistent and I can debug correct time period to view logs . The servers from where we are forwarding the logs is also in GMT time as far as I know. Time-mockup:

gsonal03 · ‎03-08-2019

Unfortunately, we did not find reference to blocked queues in splunkd log. The only reference to queues occurred during shutdown messages. In the given timeframe where the logs went missing/delayed, we see info logs stating "TcpOutputProc - Connected to idx", nothing else. We also wanted to review metrics.log based on the article https://docs.splunk.com/Documentation/Splunk/7.2.4/Troubleshooting/Aboutmetricslog, but the logs from last week are no longer available. Now, I will need to wait until the problem reoccurs.

gsonal03 · ‎03-07-2019

Thanks, I will review this with internal team and get back. Should we also change the throttle limits of the forwarder - I see such suggestions in another community article - https://answers.splunk.com/answers/579305/delayed-log-ingestion.html? The application does send multiple logs within span of few seconds. Not sure if this also causes the queue to get filled, if such is the case.

gsonal03 · ‎03-07-2019

The difference is big, so it is "delayed log ingestion problem". 2019-02-28 09:38:25.402 2019-02-28 09:38:26 0.598 2019-02-28 09:38:10.868 2019-02-28 09:38:11 0.132 2019-02-28 09:37:55.963 2019-02-28 09:37:56 0.037 2019-02-28 09:37:41.743 2019-02-28 09:37:43 1.257 2019-02-28 04:39:40.263 2019-02-28 09:42:14 **18153.737** 2019-02-28 04:39:11.629 2019-02-28 09:41:47 **18155.371** 2019-02-28 04:38:58.574 2019-02-28 09:41:36 **18157.426** 2019-02-28 04:37:08.356 2019-02-28 09:39:44 **18155.644** It occurred twice in past 2 weeks, so should we be looking at some underlying issue in splunk forwarder? Or are there any guidelines to prevent this issue to occur in production?

gsonal03 · ‎03-07-2019

Thanks for your inputs. We reviewed the application logs and in the small timeframe of test run, where logs are missing, we dont see any exceptions. The 200 success logs itself are missing (rather delayed, as per current findings).

gsonal03 · ‎03-07-2019

We did not find any parser error for the given timeframe. Although, the team did notice that some of the logs were "forwarded" delayed by 3 hours, while others were forwarded at the same time. Therefore, even in splunk we see some of the missing logs available outside the actual tomcat server test run timeframe. For e.g. In tomcat all the logs are seen between 4:30 to 4:35 pm, while in splunk the log statements are divided like below: 4:32 to 4:35 p.m. - statements 1-4, 10-15 6:30 - statements 5-8, and so on..

gsonal03 · ‎03-05-2019

It's the same log file from where some logs are being read and some are lost. So, it doesn't look like permission issue at this point.

gsonal03 · ‎03-05-2019

I will inform my organisation infra team to try these commands out. I don't have direct access to the server at this point. In general, the observation is for a given timeframe also (as small as 30 seconds), some logs are visible, while next few lines are not visible in splunk (I am assuming they were not indexed correctly).

gsonal03 · ‎03-05-2019

We have a tomcat application generating logs on the server, which are then indexed and available in Splunk. For the past 3 weeks, we have noticed that abruptly, some logs are not visible under Splunk. We also exported the actual Tomcat logs from the server, and see that the original log file contains all logs as expected, but Splunk is missing several statements. What should be our troubleshooting approach for such an issue? I am unable to find any existing issues like this in open source community articles.

Posts	16
Solutions	0
Karma Given	0
Karma Received	0
Member Since	‎03-05-2019

Online Status	Offline
Date Last Visited	‎06-05-2020 02:04 AM

Splunk forwarder delays only Concurrently generate...

Splunk indexing delays showing as negative.

What time is displayed in raw splunk logs

Why are our Splunk indexes not showing all log ent...

Re: Splunk indexing delays showing as negative.

Re: Splunk indexing delays showing as negative.

Re: Splunk indexing delays showing as negative.

Splunk forwarder delays only Concurrently generate...

Re: Splunk indexing delays showing as negative.

Splunk indexing delays showing as negative.

Re: What time is displayed in raw splunk logs

What time is displayed in raw splunk logs

Re: Why are our Splunk indexes not showing all log...

Re: Why are our Splunk indexes not showing all log...

Re: Why are our Splunk indexes not showing all log...

Re: Why are our Splunk indexes not showing all log...

Re: Why are our Splunk indexes not showing all log...

Re: Why are our Splunk indexes not showing all log...

Re: Why are our Splunk indexes not showing all log...

Why are our Splunk indexes not showing all log ent...