New Cluster Peer Caused Search Head to EXPLODE!

[joshualemoine]

Okay, hopefully the Title was juicy enough to pull you in. Now to the facts.

We added 10 new Indexers to our Index Cluster over a week ago, and they seemed to be working without error. Now, seven days later, we received this error message on our Search Head, along with all searches/reports/alerts/dashboards failing:

lpec51409spkix35.yourdomain.net Steamed search execute failed because: Error in 'litsearch' command: Your Splunk license expired or you have exceeded your license limit too many times

Now, our license is a "no fault" 5TB license, and our daily reports show us hovering around 4TB Indexed per day, for some time now, including the days up to, and surrounding this issue.

The message that stands out in splunkd.log on this Indexer is, and the solution may seem obvious, but I want your folks opinion as it requires an emergency change to fix it, and I want to get it right the first time:

LMTracker - failed to send rows, reason='WARN: path=/masterlm/usage: Signature mismatch between license slave=10.x.x.x and this License Master. Please make sure that the pass4SymmKey setting in server.conf, under [general], is the same for the License Master and all its slaves from ip=10.x.x.x'

When I check the value of pass4SymmKey in server.conf, the encrypted value does appear to be different on this Indexer than on the License Master. Is that it? Is it because this Indexer can't communicate with the LM and it's triggering a Search Shutdown b/c it's Indexed more than the trial/free version allows?

If this is the issue, my plan is to put the raw text passkey into server.conf on the Indexer and restart Splunk, but the problem is, for whatever reason, I have to downgrade to Splunk 6.6 first, make the change to the config, start Splunk, let it encrypt the value and check in with the cluster master and license master, then upgrade back to 7.2.1. It just won't encrypt the key right if I do it at 7.2.1. (I've found this out from experience in our deployment.

Thanks for the help!
-J

P.S. Will downgrading, then upgrading, corrupt any of the Indexed data already on the Indexer?

[Anonymous]

New guy here, but would it not be easier to set all servers to free licensing.
This will take down search and login.

Then delete the lic file, an re-upload it to the license master.
And the log on each one and ad the new"old" licensing master?

Would not this do the trick?

[joshualemoine]

Thanks for the response! I'm not sure I'm following the your train of thought. The real problem was that this single new Indexer didn't have the right pass4SymmKey, and the only way to get it the right now, was to paste the plain text in it, and restart Splunk. However, restarting it in the 7.2.1 environment produces an encrypted passkey that is much longer, and incorrect than the right one, so downgrading to 6.6 on just this single Indexer, and then changing the pass4SymmKey, restarting splunk, letting it encrypt, then upgrade back to 7.2.1, was my fix. It took about 30 minutes, but it didn't take the Search Head down during the change, so that was a plus.

The more I chew on your answer, are you saying that by deleting the license file on the LM, and re-uploading it, that the Indexer would somehow right itself and check-in?

Again, thanks for your input! I Always appreciate other's theories and ideas!

[ddrillic]

You should contact Splunk Support for such a case.

[joshualemoine]

I did open a case with Splunk Support. I even gave them my theory and a diag. I heard from them once when they asked me "if this is the first time it happened," but nothing after that. I just went ahead with the emergency change and fixed it.

Support is always the first line of defense if there's something I can't figure out and it needs attention fast. After that, I turn here.

Anyways, thanks for the comment. It was received as positive intent.