Re: Universal forwarder data stops logging at the ...

dunyaelbasan · ‎10-01-2020

We are using Splunk Light Version 8.0.0 but have discovered recently that Splunk seems to stop logging for a few days once a new month starts.

I've attached the splunkd.logs from 00:00.

09-30-2020 00:00:00.027 +0300 INFO LMStackMgr - finished rollover, new lastRolloverTime=1601413200
09-30-2020 00:00:27.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:00:35.446 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:00:48.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:00:59.027 +0300 INFO LMSlaveInfo - Detected that masterTimeFromSlave(Tue Sep 29 23:59:58 2020) < lastRolloverTime(Wed Sep 30 00:00:00 2020), meaning that the master has already rolled over. Ignore slave persisted usage.
09-30-2020 00:00:59.554 +0300 WARN TcpOutputProc - Cooked connection to ip=10.210.50.94:9997 timed out
09-30-2020 00:01:09.028 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:01:30.030 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:01:35.480 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:01:51.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:01:59.299 +0300 WARN TcpOutputProc - Cooked connection to ip=10.210.50.94:9997 timed out
09-30-2020 00:02:12.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:02:33.031 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:02:35.492 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:02:35.504 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:02:54.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:03:15.028 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:03:35.504 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:03:36.028 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:03:57.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:03:58.786 +0300 WARN TcpOutputProc - Cooked connection to ip=10.210.50.94:9997 timed out
09-30-2020 00:04:19.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:04:35.516 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:04:40.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:05:02.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:05:23.028 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:05:35.528 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:05:44.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:06:05.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:06:27.026 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:06:28.314 +0300 WARN TcpOutputProc - Cooked connection to ip=10.210.50.94:9997 timed out
09-30-2020 00:06:35.540 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:06:48.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:07:18.028 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:07:28.094 +0300 WARN TcpOutputProc - Cooked connection to ip=10.210.50.94:9997 timed out
09-30-2020 00:07:35.551 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:07:40.031 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:08:02.028 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:08:24.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:08:35.563 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:08:45.033 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:09:07.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:09:27.789 +0300 WARN TcpOutputProc - Cooked connection to ip=10.210.50.94:9997 timed out

dunyaelbasan · ‎10-13-2020

Hi @gcusello ,

Time formats are same for other 2 hosts.

For this month problematic 2 of them are started to log after 10 days. (I think its a poof that issue is related to time format. Because we are in the 10th month and data flow started after 10 days. I believe if we can configure time format correctly we can fix the issue.)

gcusello · ‎10-13-2020

Hi @dunyaelbasan,

as I said the problem is probably related to the time format that's in european format (dd/mm/yyyy), instead Splunk uses by default the american format (mm/dd/yyyy), you can confirm this if you search your logs of 1st of october on the 10th of january.

To solve this, you have to add to your sourcetype (in props.conf) the TIME_FORMAT option, something like this:

TIME_FORMAT = %d-%m-%Y %H:%M:%S.%3N

Then I saw that the second logs you sent are different from the first one (also TIME_FORMAT), this means that you should analyze better your sources and define the pecifications of each one and configure a dedicated sourcetype for each.

Ciao.

Giuseppe

dunyaelbasan · ‎10-02-2020

Hi @gcusello

Thanks again for the quick reply. Unfortunately, it didn't work for me. Every month we cant get data for 2 Hosts (not all of them) during aprox. 7 days.

gcusello · ‎10-02-2020

Hi @dunyaelbasan,

sorry but I don't understand: why you "cant get data for 2 Hosts (not all of them) during aprox. 7 days."?

can you descrive more why?

Maybe the other hosts have a differente timestamp format?

If they have the same timestamp format in all the events with that sourcetype, you can use my hint, if they have a different timestamp format, you have to identify sources with the same time format and use a sourcetype for each of them.

Ciao.

Giuseppe

dunyaelbasan · ‎10-01-2020

Thanks for reply

Here is my log format:

[01-10-2020 04:36:00.006] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:36:00.023] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 04:39:00.001] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:39:00.016] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 04:42:00.009] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:42:00.028] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 04:45:00.000] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - tryFailedPayments

[01-10-2020 04:45:00.010] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:45:00.066] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 04:45:11.888] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - tryFailedPayments

[01-10-2020 04:48:00.001] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:48:00.025] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 04:51:00.008] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:51:00.031] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 04:54:00.010] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:54:00.025] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 04:57:00.000] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:57:00.019] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 05:00:00.000] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 05:00:00.009] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - tryFailedPayments

[01-10-2020 05:00:00.009] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - preAuthReverse

[01-10-2020 05:00:00.020] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 05:00:10.556] [INFO ] -- com.turkcell.masrafim.exception.InvalidTokenHandler - hasError - AF 401 or 403 status code received, system will try to retrieve valid token

[01-10-2020 05:00:10.556] [INFO ] -- com.turkcell.masrafim.exception.InvalidTokenHandler - handleError - Attempting to retrieve AF token

[01-10-2020 05:00:10.555] [INFO ] -- com.turkcell.masrafim.exception.InvalidTokenHandler - hasError - AF 401 or 403 status code received, system will try to retrieve valid token

[01-10-2020 05:00:10.557] [INFO ] -- com.turkcell.masrafim.exception.InvalidTokenHandler - handleError - Attempting to retrieve AF token

[01-10-2020 04:33:00.003] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:33:00.030] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob