Splunk documentation ("Harden your KV store port") states "we recommend that you secure your environment by restricting KV store access to your port" but there doesn't seem to be any documentation stating any of the following:
As to the first point, I presume the risks are potential exfiltration of data and/or alteration of the kvstore - but that goes to the second point: why isn't integral security adequate? Is MongoDB security broken? Are connections encrypted? how is a connection authenticated? The KV store documents don't mention any of this.
Based on the MongoDB documentation I presume the recommended hardening method is iptables, but Splunk docs don't mention this either.
In other words, what is the basis of this recommendation? More info/documentation is needed.
You may be reading too much into the two-paragraph document. It was written by Splunk as a suggestion to their entire customer base.
In essence, by default your entire network has access to port 8191. If you allow external connections to your network, then they have access to that port. Splunk doesn't know what you have in your KV store, or who can access your network.
Splunk recommends you limit access to that port on that box to whatever has a valid need to access that particular box, and specifically to access the KV store.
They've brought your attention to the subject, so you can make reasonable decisions based upon your own architecture. The specific method will depend on your infrastructure, business requirements, operating system, and so on.
I apologize if I was insufficiently explicit about what, specifically, you were overcomplicating.
I'm going to give you at least a paragraph for each question, including one for the cluster of them that wasn't numbered. If any of these are difficult to understand, then please reply back with specifically what is confusing you about that answer.
Let's start with your question 3: "What the appropriate methods would be to harden [the port]?"
The last sentence of the recommendation specifically says ...
Splunk: "For search head clustering, you should open the port only to other members of the cluster so that other members can replicate KV store data."
That's a pretty explicit and simple recommendation. Unless you have a good reason NOT to do that, do that.
Now let me go back to number 1: "What is the risk of NOT hardening the port?"
...which you answered yourself.
You: "... I presume the risks are potential exfiltration of data and/or alteration of the kvstore"
Now I'll add a whole bunch more wording.
If your particular configuration and architecture does not in any way expose that port to attackers, then nothing is required. However, it is your responsibility to understand your own architecture and make that call. Splunk did not design your system, you did.
They are merely calling to your attention that if that port is unsecured and can be accessed by arbitrary compromised systems, then your KV store is potentially exposed, for exfiltration or infiltration.
Proceeding with Question 2: "What, if any, integral security is included with the KV store?"
The structure of the specific calls were described on the page you pointed to. That structure includes the user and password. http://docs.splunk.com/Documentation/Splunk/latest/Admin/TroubleshootKVstore
Various other potential calls are spread throughout the Splunk documentation, and many of the external specifics are going to depend on your operating system and network architecture, which we'll discuss in a moment.
Proceeding with question 4. "Why isn't integral security adequate?"
Ummm. WHICH integral security? How "secure" represents "adequate" for your business use case? How confidential is the data? How important is the data? How critical is the data if it is lost? What is the potential risk? What is your network configuration? What are the goals of a hacker going to be when they attack your organization? What kinds of attacks do you need to defend against?
Your subquestions on this one: Are connections encrypted? How is a connection authenticated?
These are operating system and network architecture questions. Splunk has no idea whether you choose to encrypt network traffic across your network.
If you would LIKE to harden any and all connections used by Splunk across your entire network, then you can reference the suggestions here: https://wiki.splunk.com/Community:DeployHardenedSplunk
Unfortunately not. I was hoping for additional answers.
I don't think I was reading too much into the document; there is nothing to read into it, and that's the problem.
Splunk is making a recommendation with no justification (indication of risk), best practices, or any guidance at all. The intent of the document is unclear and its purpose is even murkier.