topic Re: Universal forwarder data stops logging at the beginning of the month in Reporting

Universal forwarder data stops logging at the beginning of the month

dunyaelbasan — Thu, 01 Oct 2020 08:17:28 GMT

We are using Splunk Light Version 8.0.0 but have discovered recently that Splunk seems to stop logging for a few days once a new month starts.

I've attached the splunkd.logs from 00:00.

09-30-2020 00:00:00.027 +0300 INFO LMStackMgr - finished rollover, new lastRolloverTime=1601413200
09-30-2020 00:00:27.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:00:35.446 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:00:48.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:00:59.027 +0300 INFO LMSlaveInfo - Detected that masterTimeFromSlave(Tue Sep 29 23:59:58 2020) < lastRolloverTime(Wed Sep 30 00:00:00 2020), meaning that the master has already rolled over. Ignore slave persisted usage.
09-30-2020 00:00:59.554 +0300 WARN TcpOutputProc - Cooked connection to ip=10.210.50.94:9997 timed out
09-30-2020 00:01:09.028 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:01:30.030 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:01:35.480 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:01:51.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:01:59.299 +0300 WARN TcpOutputProc - Cooked connection to ip=10.210.50.94:9997 timed out
09-30-2020 00:02:12.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:02:33.031 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:02:35.492 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:02:35.504 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:02:54.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:03:15.028 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:03:35.504 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:03:36.028 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:03:57.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:03:58.786 +0300 WARN TcpOutputProc - Cooked connection to ip=10.210.50.94:9997 timed out
09-30-2020 00:04:19.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:04:35.516 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:04:40.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:05:02.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:05:23.028 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:05:35.528 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:05:44.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:06:05.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:06:27.026 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:06:28.314 +0300 WARN TcpOutputProc - Cooked connection to ip=10.210.50.94:9997 timed out
09-30-2020 00:06:35.540 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:06:48.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:07:18.028 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:07:28.094 +0300 WARN TcpOutputProc - Cooked connection to ip=10.210.50.94:9997 timed out
09-30-2020 00:07:35.551 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:07:40.031 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:08:02.028 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:08:24.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:08:35.563 +0300 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.214.120.71_8089_10.214.120.71_radian01_CD4B2B7B-B69D-4097-B528-0F7B136F6DF1
09-30-2020 00:08:45.033 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:09:07.027 +0300 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value
09-30-2020 00:09:27.789 +0300 WARN TcpOutputProc - Cooked connection to ip=10.210.50.94:9997 timed out

Re: Universal forwarder data stops logging at the beginning of the month

gcusello — Thu, 01 Oct 2020 09:24:43 GMT

Hi @dunyaelbasan,

probably the problem is that there's a wrong configuration of Timestamp.

In other words you probably have in your logs a date in European format (dd/mm/yyyy hh.mm.ss), instead Splunk read it in American format (mm/dd/yyyy hh:mm:ss), infact Splunk correctly read your timestamp when day and month are the same or when there's non dubt (e.g. days greater than 12).

So your logs are indexed with a wrong date (e.g. 1st of September is read as 9th of January).

Verify your TIME_FORMAT or share an example of your log.

If you want an help in this check you should share some log example (from the source not from splunkd) to check your TIME_FORMAT.

Ciao.

Giuseppe

Re: Universal forwarder data stops logging at the beginning of the month

dunyaelbasan — Thu, 01 Oct 2020 18:05:10 GMT

Thanks for reply

Here is my log format:

[01-10-2020 04:36:00.006] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:36:00.023] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 04:39:00.001] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:39:00.016] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 04:42:00.009] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:42:00.028] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 04:45:00.000] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - tryFailedPayments

[01-10-2020 04:45:00.010] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:45:00.066] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 04:45:11.888] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - tryFailedPayments

[01-10-2020 04:48:00.001] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:48:00.025] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 04:51:00.008] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:51:00.031] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 04:54:00.010] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:54:00.025] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 04:57:00.000] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:57:00.019] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 05:00:00.000] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 05:00:00.009] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - tryFailedPayments

[01-10-2020 05:00:00.009] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - preAuthReverse

[01-10-2020 05:00:00.020] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob

[01-10-2020 05:00:10.556] [INFO ] -- com.turkcell.masrafim.exception.InvalidTokenHandler - hasError - AF 401 or 403 status code received, system will try to retrieve valid token

[01-10-2020 05:00:10.556] [INFO ] -- com.turkcell.masrafim.exception.InvalidTokenHandler - handleError - Attempting to retrieve AF token

[01-10-2020 05:00:10.555] [INFO ] -- com.turkcell.masrafim.exception.InvalidTokenHandler - hasError - AF 401 or 403 status code received, system will try to retrieve valid token

[01-10-2020 05:00:10.557] [INFO ] -- com.turkcell.masrafim.exception.InvalidTokenHandler - handleError - Attempting to retrieve AF token

[01-10-2020 04:33:00.003] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB STARTED - failedRequestRetryJob

[01-10-2020 04:33:00.030] [INFO ] -- com.turkcell.masrafim.scheduler.SchedulerJobBean - logJobInfo - JOB ENDED - failedRequestRetryJob