After a long-overdue upgrade from 6.x to 7.1.3 -- this release it the latest one supported by my vendor, who interoperates with Splunk -- I have a problem. The search head no longer works with the in...
See more...
After a long-overdue upgrade from 6.x to 7.1.3 -- this release it the latest one supported by my vendor, who interoperates with Splunk -- I have a problem. The search head no longer works with the indexers. On the search head: The full message in splunkd.log is: "Global key files are invalid. This server cannot distribute searches to other servers." In Settings » Distributed search » Search peers , we have error messages: Error [00000100] Instance name "<deleted>" REST interface to peer is not responding. Check var/log/splunk/splunkd_access.log on the peer. Last Connect Time:2020-09-14T20:04:01.000+00:00; Failed 1 out of 1 times. If I delete the distributed search head and attempt to re-validate it, I get the error: Encountered the following error while trying to save: Invalid action for this internal handler (handler: distsearch-peer, supported: list|edit|remove|_reload|new|disable|enable|doc, wanted: create). The only way I've found to re-add the search peer is to restart Splunk on the search head. Also of note on the search head: because of changes by my vendor -- as far as I can tell -- when I install the upgraded Splunk, the vendor automatically restores the old file $splunk/etc/auth//distServerKeys/trusted.pem. As a result, again as far as I can tell, when I start Splunk for the first time, the file $splunk/etc/auth//distServerKeys/private.pem is never generated on the search head. The search peers, on the other hand, do have both files. Also in splunkd.log, I see messages such as: DistributedPeer - Peer:https://x.x.x.x:y Key problems, see internal logs with no indication of where these "internal logs" can be found. I also see Bundle Replication: Problem replicating config (bundle) to search peer ' x.x.x.x:y ', HTTP response code 401 (HTTP/1.1 401 Unauthorized). call not properly authenticated On the search peers: The search peer logs do not currently show any particular issue. The splunkd.log shows, on both indexers (search peers): WARN HTTPAuthManager - Token not specified in Authorization: Splunk <token> header and in splunkd_access.log, POST /services/receivers/bundle/<search head address> HTTP/1.0" 401 148 - - - 0ms which provides no useful information. On the search peeers, the directory $splunk/etc/auth/distServerKeys/<search head name> has an exact copy of the file $splunk/etc/auth//distServerKeys/trusted.pem. Questions: Why does this fail? Is this due to $splunk/etc/auth//distServerKeys/trusted.pem being present on the search head, with some incorrect key information? What does "Global key files are invalid" mean, and where can I find further information about how to fix them? I welcome other suggestions -- as this includes suggestions for the right questions to ask.