I am very thankful for your quick inputs. I am looking forward to raise a case to Splunk support team. Once the issue gets fixed, i will update the resolving steps here. Thank you! ... View more

We have not installed any new service. The environment was fine until yesterday morning. we have not performed any activity recently. ... View more

Are this can be ignored? 12-10-2018 06:01:23.883 +0000 INFO TailReader - Ignoring file '/var/log/flume-ng/flume-cmf-flume-AGENT-sfiappnwh040.statefarm-dss.com.log' due to: binary 12-10-2018 06:01:23.883 +0000 WARN FileClassifierManager - The file '/var/log/flume-ng/flume-cmf-flume-AGENT-sfiappnwh040.statefarm-dss.com.log' is invalid. Reason: binary 12-10-2018 06:01:23.734 +0000 INFO TailReader - Ignoring file '/var/log/flume-ng/flume-cmf-flume-AGENT-sfiappnwh040.statefarm-dss.com.log' due to: binary 12-10-2018 06:01:23.734 +0000 WARN FileClassifierManager - The file '/var/log/flume-ng/flume-cmf-flume-AGENT-sfiappnwh040.statefarm-dss.com.log' is invalid. Reason: binary 12-10-2018 06:01:23.655 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/var/log/hue/metrics-hue_server/metrics.log'. 12-10-2018 06:01:23.655 +0000 INFO TailReader - Ignoring file '/var/log/hue/metrics-hue_server/tmpZnYoeF' due to: failed_stat 12-10-2018 06:01:23.655 +0000 WARN FileClassifierManager - The file '/var/log/hue/metrics-hue_server/tmpZnYoeF' is invalid. Reason: failed_stat ... View more

Hi , After stopping the antivirus scanning process server performance was improved. I am able to get the some search results(partially) in splunk but splunkd.log showing so many errors 12-10-2018 00:50:19.162 -0500 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Mon Dec 10 00:50:15 2018). Context: source::/data5/yarn/container-logs/application_1537825532257_22021/container_e42_1537825532257_22021_01_000002/stdout|host::sfiappnwh021.statefarm-dss.com|stdout-too_small|1431345 12-10-2018 00:50:19.162 -0500 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Mon Dec 10 00:50:15 2018). Context: source::/data5/yarn/container-logs/application_1537825532257_22021/container_e42_1537825532257_22021_01_000002/stdout|host::sfiappnwh021.statefarm-dss.com|stdout-too_small|1431345 12-10-2018 00:50:19.162 -0500 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Mon Dec 10 00:50:15 2018). Context: source::/data5/yarn/container-logs/application_1537825532257_22021/container_e42_1537825532257_22021_01_000002/stdout|host::sfiappnwh021.statefarm-dss.com|stdout-too_small|1431345 12-10-2018 00:50:19.162 -0500 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Mon Dec 10 00:50:15 2018). Context: source::/data5/yarn/container-logs/application_1537825532257_22021/container_e42_1537825532257_22021_01_000002/stdout|host::sfiappnwh021.statefarm-dss.com|stdout-too_small|1431345 12-10-2018 00:50:19.162 -0500 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Mon Dec 10 00:50:15 2018). Context: source::/data5/yarn/container-logs/application_1537825532257_22021/container_e42_1537825532257_22021_01_000002/stdout|host::sfiappnwh021.statefarm-dss.com|stdout-too_small|1431345 12-10-2018 00:50:19.162 -0500 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Mon Dec 10 00:50:15 2018). Context: source::/data5/yarn/container-logs/application_1537825532257_22021/container_e42_1537825532257_22021_01_000002/stdout|host::sfiappnwh021.statefarm-dss.com|stdout-too_small|1431345 12-10-2018 00:50:19.162 -0500 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Mon Dec 10 00:50:15 2018). Context: source::/data5/yarn/container-logs/application_1537825532257_22021/container_e42_1537825532257_22021_01_000002/stdout|host::sfiappnwh021.statefarm-dss.com|stdout-too_small|1431345 12-10-2018 00:50:19.162 -0500 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Mon Dec 10 00:50:15 2018). Context: source::/data5/yarn/container-logs/application_1537825532257_22021/container_e42_1537825532257_22021_01_000002/stdout|host::sfiappnwh021.statefarm-dss.com|stdout-too_small|1431345 12-10-2018 00:50:19.162 -0500 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Mon Dec 10 00:50:15 2018). Context: source::/data5/yarn/container-logs/application_1537825532257_22021/container_e42_1537825532257_22021_01_000002/stdout|host::sfiappnwh021.statefarm-dss.com|stdout-too_small|1431345 12-10-2018 00:50:19.162 -0500 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Mon Dec 10 00:50:03 2018). Context: source::/data5/yarn/container-logs/application_1537825532257_22021/container_e42_1537825532257_22021_01_000002/stdout|host::sfiappnwh021.statefarm-dss.com|stdout-too_small|1431345 12-10-2018 00:50:19.676 -0500 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 32773 - data_source="/data5/yarn/container-logs/application_1537825532257_22021/container_e42_1537825532257_22021_01_000002/syslog", data_host="sfiappnwh021.statefarm-dss.com", data_sourcetype="syslog" 12-10-2018 00:50:21.960 -0500 WARN HandleJobsDataProvider - Provenance argument was in an invalid format. 12-10-2018 00:50:21.998 -0500 WARN AdminManager - Handler 'summarization' has not performed any capability checks for this operation (requestedAction=list, customAction="", item=""). This may be a bug. 12-10-2018 00:50:24.156 -0500 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 34066 - data_source="/data4/yarn/container-logs/application_1537825532257_21753/container_e42_1537825532257_21753_01_016922/syslog", data_host="sfiappnwh018.statefarm-dss.com", data_sourcetype="syslog" 12-10-2018 00:50:29.069 -0500 WARN HandleJobsDataProvider - Provenance argument was in an invalid format. 12-10-2018 00:50:29.106 -0500 WARN AdminManager - Handler 'summarization' has not performed any capability checks for this operation (requestedAction=list, customAction="", item=""). This may be a bug. 12-10-2018 00:50:30.096 -0500 WARN HandleJobsDataProvider - Provenance argument was in an invalid format. 12-10-2018 00:50:30.552 -0500 WARN AdminManager - Handler 'summarization' has not performed any capability checks for this operation (requestedAction=list, customAction="", item=""). This may be a bug. ... View more

Hello, The indexer having 36 TB free space and entire application having no no network issues. ... View more

Hi, The below is the splunk universal forwarder log. 10.61.1.81 is the master node. I am afraid 10.61.1.81 indexer too. 12-10-2018 04:16:20.474 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.61.1.78_8089_sfiappnwh027.statefarm-dss.com_sfiappnwh027.statefarm-dss.com_E56FB3B9-46F9-4430-8F91-AA8496ED0C2A 12-10-2018 04:17:20.475 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.61.1.78_8089_sfiappnwh027.statefarm-dss.com_sfiappnwh027.statefarm-dss.com_E56FB3B9-46F9-4430-8F91-AA8496ED0C2A 12-10-2018 04:17:57.809 +0000 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 200 seconds. 12-10-2018 04:18:20.478 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.61.1.78_8089_sfiappnwh027.statefarm-dss.com_sfiappnwh027.statefarm-dss.com_E56FB3B9-46F9-4430-8F91-AA8496ED0C2A 12-10-2018 04:19:20.482 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.61.1.78_8089_sfiappnwh027.statefarm-dss.com_sfiappnwh027.statefarm-dss.com_E56FB3B9-46F9-4430-8F91-AA8496ED0C2A 12-10-2018 04:19:37.826 +0000 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 300 seconds. 12-10-2018 04:20:20.485 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.61.1.78_8089_sfiappnwh027.statefarm-dss.com_sfiappnwh027.statefarm-dss.com_E56FB3B9-46F9-4430-8F91-AA8496ED0C2A 12-10-2018 04:20:27.834 +0000 WARN TcpOutputProc - Raw connection to ip=10.61.1.81:9997 timed out 12-10-2018 04:20:27.834 +0000 INFO TcpOutputProc - Ping connection to idx=10.61.1.81:9997 timed out. continuing connections 12-10-2018 04:21:17.842 +0000 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 400 seconds. 12-10-2018 04:21:20.489 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.61.1.78_8089_sfiappnwh027.statefarm-dss.com_sfiappnwh027.statefarm-dss.com_E56FB3B9-46F9-4430-8F91-AA8496ED0C2A 12-10-2018 04:22:20.492 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.61.1.78_8089_sfiappnwh027.statefarm-dss.com_sfiappnwh027.statefarm-dss.com_E56FB3B9-46F9-4430-8F91-AA8496ED0C2A 12-10-2018 04:22:57.859 +0000 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 500 seconds. 12-10-2018 04:23:20.495 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.61.1.78_8089_sfiappnwh027.statefarm-dss.com_sfiappnwh027.statefarm-dss.com_E56FB3B9-46F9-4430-8F91-AA8496ED0C2A 12-10-2018 04:24:20.498 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.61.1.78_8089_sfiappnwh027.statefarm-dss.com_sfiappnwh027.statefarm-dss.com_E56FB3B9-46F9-4430-8F91-AA8496ED0C2A 12-10-2018 04:24:37.876 +0000 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 600 seconds. ... View more

I am also seeing below logs info continuously repeated(look for data_source line) AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded - data_source="/var/log/hive/metrics-hivemetastore/metrics.log", data_host="sfiappnwh026.statefarm-dss.com", data_sourcetype="metrics" 12-09-2018 02:01:52.337 -0500 WARN DateParserVerbose - A possible timestamp match (Tue Dec 24 06:51:04 2019) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source::/var/log/hive/metrics-hivemetastore/metrics.log|host::sfiappnwh026.statefarm-dss.com|metrics|83303 12-09-2018 02:01:52.338 -0500 WARN AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded - data_source="/var/log/hive/metrics-hivemetastore/metrics.log", data_host="sfiappnwh026.statefarm-dss.com", data_sourcetype="metrics" 12-09-2018 02:02:24.493 -0500 INFO TailReader - ...continuing. 12-09-2018 02:02:24.503 -0500 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sun Dec 9 01:45:01 2018). Context: source::/var/log/rabbitmq_queue_size__check.out|host::sfisvlnwh007.statefarm-dss.com|breakable_text|233395 12-09-2018 02:03:01.494 -0500 INFO TailReader - Could not send data to output queue (parsingQueue), retrying... 12-09-2018 02:03:27.099 -0500 WARN PeriodicReapingTimeout - Spent 85084ms updating search-related banner messages 12-09-2018 02:03:27.105 -0500 INFO PipelineComponent - MetricsManager:probeandreport() took longer than seems reasonable (96007 milliseconds) in callbackRunnerThread. Might indicate hardware or splunk limitations. 12-09-2018 02:03:28.717 -0500 WARN AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded - data_source="/var/log/hive/metrics-hivemetastore/metrics.log", data_host="sfiappnwh026.statefarm-dss.com", data_sourcetype="metrics" 12-09-2018 02:03:28.717 -0500 WARN AggregatorMiningProcessor - Changing breaking behavior for event stream because MAX_EVENTS (256) was exceeded without a single event break. Will set BREAK_ONLY_BEFORE_DATE to False, and unset any MUST_NOT_BREAK_BEFORE or MUST_NOT_BREAK_AFTER rules. Typically this will amount to treating this data as single-line only. - data_source="/var/log/hive/metrics-hivemetastore/metrics.log", data_host="sfiappnwh026.statefarm-dss.com", data_sourcetype="metrics" 12-09-2018 02:03:28.792 -0500 INFO TailReader - ...continuing. 12-09-2018 02:03:28.872 -0500 WARN AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded - data_source="/var/log/hive/metrics-hivemetastore/metrics.log", data_host="sfiappnwh026.statefarm-dss.com", data_sourcetype="metrics" 12-09-2018 02:03:28.872 -0500 WARN AggregatorMiningProcessor - Changing breaking behavior for event stream because MAX_EVENTS (256) was exceeded without a single event break. Will set BREAK_ONLY_BEFORE_DATE to False, and unset any MUST_NOT_BREAK_BEFORE or MUST_NOT_BREAK_AFTER rules. Typically this will amount to treating this data as single-line only. - data_source="/var/log/hive/metrics-hivemetastore/metrics.log", data_host="sfiappnwh026.statefarm-dss.com", data_sourcetype="metrics" 12-09-2018 02:03:33.795 -0500 INFO TailReader - Could not send data to output queue (parsingQueue), retrying... ... View more

The above log is from Splunk Master. Yesterday i restarted the splunk master, but after 1 hour every thing became bad as i said above. ... View more

HI, I understood that peek_index_throughput means the amount of data does travel through the index pipeline. This changes time to time depends on the system performance, network etc etc. Thanks for your quick help on this. I used query-2 for the splunk license usage. ... View more

HI, One final question. Can you just elaborate what is mean by per_index_throughput? Is it the amount data passed the index pipeline ? If yes how it is different with query-2. Thanks ... View more

Hello, Thanks for your reply. While doing the DMC setup there is a warning coming up , you can refer the same in attached pic. I would like to know which is the best query for splunk license usage calculation. So that i can go with the Renewal according to the usage. ... View more

I tried this. It didn't work with my dash board. ... View more

Still am getting event count change on every run. Don't know the exact problem. I am observing this kind of change first time. our data is a real time log which will generate from mobile application. May be different time zones causing this issue. Thanks. ... View more

Still am getting event count change on every run. Don't know the exact problem. I am observing this kind of change first time. our data is a real time log which will generate from mobile application. ... View more

Thanks for your reply..Let me try it out.will get you back with the result. ... View more

Might be Your company blocking that port. That's why you are unable to get the login page. ... View more

Hi I used the below query sourcetype="mylogdata" type="data" earliest=-1d@h-5 latest=-1d@h-3 HTTP_Status="202" |timechart span=1h count The below output i got _time count 2018-06-11 07:00 34 Just fyi, i have extracted fileds. What did i take over from the above query? I am observing event count change for almost all the hours. Also Please let me know the meaning of the below line earliest=-1d@h-5 latest=-1d@h-3 ... View more