Okay, hopefully the Title was juicy enough to pull you in. Now to the facts.
We added 10 new Indexers to our Index Cluster over a week ago, and they seemed to be working without error. Now, seven days later, we received this error message on our Search Head, along with all searches/reports/alerts/dashboards failing:
lpec51409spkix35.yourdomain.net Steamed search execute failed because: Error in 'litsearch' command: Your Splunk license expired or you have exceeded your license limit too many times
Now, our license is a "no fault" 5TB license, and our daily reports show us hovering around 4TB Indexed per day, for some time now, including the days up to, and surrounding this issue.
The message that stands out in splunkd.log on this Indexer is, and the solution may seem obvious, but I want your folks opinion as it requires an emergency change to fix it, and I want to get it right the first time:
LMTracker - failed to send rows, reason='WARN: path=/masterlm/usage: Signature mismatch between license slave=10.x.x.x and this License Master. Please make sure that the pass4SymmKey setting in server.conf, under [general], is the same for the License Master and all its slaves from ip=10.x.x.x'
When I check the value of pass4SymmKey in server.conf, the encrypted value does appear to be different on this Indexer than on the License Master. Is that it? Is it because this Indexer can't communicate with the LM and it's triggering a Search Shutdown b/c it's Indexed more than the trial/free version allows?
If this is the issue, my plan is to put the raw text passkey into server.conf on the Indexer and restart Splunk, but the problem is, for whatever reason, I have to downgrade to Splunk 6.6 first, make the change to the config, start Splunk, let it encrypt the value and check in with the cluster master and license master, then upgrade back to 7.2.1. It just won't encrypt the key right if I do it at 7.2.1. (I've found this out from experience in our deployment.
Thanks for the help!
-J
P.S. Will downgrading, then upgrading, corrupt any of the Indexed data already on the Indexer?
... View more