@woodcock I am having the same issue only difference is when my UF attempts again after a few minutes HandShake is achieved and all the applicable apps are pulled down.  This same pattern is happening across a significant number of endpoints.  Any idea why that would be the case?  ... View more

There is no object field anywhere in the data for: index=_audit user=* action=indexes_edit This is even with searching against the last 90 days.  Why is that?   ... View more

It is possible that there are other things going on that is causing this error than what is stated above.  Since I identified a unique root cause I wanted to share with all.  The last bullet below was what worked for me but the below bullets represents a summary of recommended steps to get to root cause for this. First verify that the size of the bundle being sent from SH is not greater than the bundle size limit setting on the SH (maxBundleSize distSearch.conf) or the Indexer (max_content_lengh server.conf) Then check for permissions/ownership errors on all the instanced by running “ls -lahR /opt/spunk | grep root” Then run ./splunk btool check Then check the CM bundle details and compare if the latest active bundle in the peers is same as the CM. Then run the top command to see if there are any resources using a significant percentage of CPU utilization over Splunk.  A new application could have been introduced that is preventing writes from taking place over a long period of time due to files being locked by other application.  This can be further verified by: Run the following on each indexer “sudo tcpdump <ipaddressofsourceSH>” then attempt to run your search from the SH and see if you see the commands coming over. If fails that there is an application that on in your environment that is preventing Splunk from doing what it need to do and you need to apply for an Splunk exceptions for the recently introduced application. ... View more

It is possible that there are other things going on that is causing this error than what is stated above.  Since I identified a unique root cause I wanted to share with all.  The last bullet below was what worked for me but the below bullets represents a summary of recommended steps to get to root cause for this. First verify that the size of the bundle being sent from SH is not greater than the bundle size limit setting on the SH (maxBundleSize distSearch.conf) or the Indexer (max_content_lengh server.conf) Then check for permissions/ownership errors on all the instanced by running “ls -lahR /opt/spunk | grep root” Then run ./splunk btool check Then check the CM bundle details and compare if the latest active bundle in the peers is same as the CM. Then run the top command to see if there are any resources using a significant percentage of CPU utilization over Splunk.  A new application could have been introduced that is preventing writes from taking place over a long period of time due to files being locked by other application.  This can be further verified by: Run the following on each indexer “sudo tcpdump <ipaddressofsourceSH” then attempt to run your search from the SH and see if you see the commands coming over. If fails that there is an application that on in your environment that is preventing Splunk from doing what it need to do and you need to apply for an Splunk exceptions for the recently introduced application. ... View more

It is possible that there are other things going on that is causing this error than what is stated above.  Since I identified a unique root cause I wanted to share with all.  The last bullet below was what worked for me but the below bullets represents a summary of recommended steps to get to root cause for this. First verify that the size of the bundle being sent from SH is not greater than the bundle size limit setting on the SH (maxBundleSize distSearch.conf) or the Indexer (max_content_lengh server.conf) Then check for permissions/ownership errors on all the instanced by running “ls -lahR /opt/spunk | grep root” Then run ./splunk btool check Then check the CM bundle details and compare if the latest active bundle in the peers is same as the CM. Then run the top command to see if there are any resources using a significant percentage of CPU utilization over Splunk.  A new application could have been introduced that is preventing writes from taking place over a long period of time due to files being locked by other application.  This can be further verified by: Run the following on each indexer “sudo tcpdump <ipaddressofsourceSH>” then attempt to run your search from the SH and see if you see the commands coming over. If fails that there is an application that on in your environment that is preventing Splunk from doing what it need to do and you need to apply for an Splunk exceptions for the recently introduced application. ... View more

It is possible that there are other things going on that is causing this error than what is stated above.  Since I identified a unique root cause I wanted to share with all.  The last bullet below was what worked for me but the below bullets represents a summary of recommended steps to get to root cause for this. First verify that the size of the bundle being sent from SH is not greater than the bundle size limit setting on the SH (maxBundleSize distSearch.conf) or the Indexer (max_content_lengh server.conf) Then check for permissions/ownership errors on all the instanced by running “ls -lahR /opt/spunk | grep root” Then run ./splunk btool check Then check the CM bundle details and compare if the latest active bundle in the peers is same as the CM. Then run the top command to see if there are any resources using a significant percentage of CPU utilization over Splunk.  A new application could have been introduced that is preventing writes from taking place over a long period of time due to files being locked by other application.  This can be further verified by: Run the following on each indexer “sudo tcpdump <ipaddressofsourceSH>” then attempt to run your search from the SH and see if you see the commands coming over. If fails that there is an application that on in your environment that is preventing Splunk from doing what it need to do and you need to apply for an Splunk exceptions for the recently introduced application. ... View more

@p_vitale wrote: The problem was the configuration about Distribuited Search into the master node, it was disabled. So if it is enabled, the indexers are happy about it and they don't give any error about "find bundles for search peer: MASTENODE". Would you please provide specifics on what you changed in the distsearch.com? ... View more