I have seen many questions about disabled due to licensing violation, but I applied a reset key and now I have this message during a search from the search head:
restricting search to internal indexes only (reason: [DISABLED_DUE_TO_GRACE_PERIOD,0])
The indexing cluster appears to be functioning and able to communicate to the license server on the search head. I am still unable to get back to normal searching after a licensing violation.
Any ideas? I will call Splunk tomorrow and ask support directly and post back.
So Splunk support never got back to me but I figured out the problem. The issue was indeed a communication problem between the indexing nodes and the license server. I was testing connectivity based on IP address and ports (seems reasonable — right?) All the nodes could reach the licensing server correctly. The issue was actually related to DNS failing and the indexing nodes could not resolve the license server address. I ensured the license server was entered into the hosts file and operations returned to normal.
I had the same error I did all that @khourihan_splun mentioned and tests were negative. to solve the problem I restarted splunkd on the indexer in question.
Take a look at your pass4SymmKey in your server.conf.
We were seeing the SH return a message of DISABLED_DUE_TO_GRACE_PERIOD
This means the SH can't connect to the LM, which makes it seem like we have a DNS problem.
I experienced this error message in a Splunk 6.0.3 cluster.
Here's how I troubleshot it:
If you can not connect, you have a DNS issue, and as @matthewchase above described, the work around. But I'd recommend fixing your DNS ! 🙂
Another test is to run this on the licenser endpoint:
curl -k -u admin:changeme https://localhost:8089/services/licenser/localslave
So Splunk support never got back to me but I figured out the problem. The issue was indeed a communication problem between the indexing nodes and the license server. I was testing connectivity based on IP address and ports (seems reasonable — right?) All the nodes could reach the licensing server correctly. The issue was actually related to DNS failing and the indexing nodes could not resolve the license server address. I ensured the license server was entered into the hosts file and operations returned to normal.