The reason this is happening is because the pass4SymmKey/shared-secret on your new member was not matching the secret of the rest of the cluster. When you ran the CLI command splunk add shcluster-member -current_member_uri https://server2:8089, an authentication token was generated on the search head server1 and put in its $HOME/.splunk directory. add shcluster-member proxies the request to one of the current members (server 2) and tries to get the current node added via the captain. This proxy process fails due to authentication, when the invalid token is used. What you will need to do is. 1. bring down the node that is being added and fails. 2. go to server.conf of that node and change its pass4SymmKey to correct shared secret. or re init the node. 3. delete the $HOME/.splunk folder so that the CLI gets a new key 4. restart splunk 5. try the add shcluster-member command again. ... View more

The splunk show shcluster-status should not be run on the deployer, it needs to be run on one of the 3 cluster members. think of the deployer as a separate entity outside the shcluster which needs to be able to communicate with the shcluster (using secret /pass4SymmKey) the init step needs to be run on the 3 members. The init step should leave a [shclustering] stanza in your server.conf with correct settings. Bootstrap then needs to be run on only 1 of the members. As mentioned before the status needs to be checked on only the members. If you still have issues if you follow the steps, please holler. ... View more

the pass4SymmKey in [general] would be used to talk to License Master. the pass4SymmKey in [general] and clustering would be used to talk to peers. the pass4SymmKey in [general] and [shclustering] would be used to talk to search heads in the search head cluster. The pass4SymmKey's would be encrypted, so it will be coming from the default config files into local config files as the encrypted value. Ignore the fact that it keeps coming back. It is a side-effect of this encryption process. Your shclustering clear text pass4SymmKey must have been "changeme" which is the same as the pass4SymmKey in the etc/system/default/server.conf/[general]/pass4SymmKey For this setup, please ensure that the 3 relevant pass4SymmKey's are exactly as they should be. If the pass4SymmKey's are different, then you need to use all 3 layers ([general], [clustering] and [shclustering] ) ... View more

cache_size is the raw # of searches that are cached in-memory in splunkd Quota Cache. It is for making quota checks faster. It is not a percentage ... View more

curl -k -u admin:changeme https://foobar:8101/services/auth/login -d username=admin -d password=changeme 2b8868fe198cf1203256e6af6515bfad curl -k -H "Authorization: Splunk 2b8868fe198cf1203256e6af6515bfad" https://foobar:8101/services/search/jobs | less succeeds... OUTPUT below 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0<?xml version="1.0" encoding="UTF-8"?> <!--This is to override browser formatting; see server.conf[httpServer] to disable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .--> <?xml-stylesheet type="text/xml" href="/static/atom.xsl"?> jobs https://foobar:8101/services/search/jobs 2014-02-11T10:13:55-08:00 Splunk opensearch:totalResults7/opensearch:totalResults opensearch:itemsPerPage0/opensearch:itemsPerPage opensearch:startIndex0/opensearch:startIndex {snipped} ... View more

partial information is not helping me completely debug the issue. need the values of the rtt field, since it is used in stats and geostats. so. can you provide me just the output of the lookup, or else... .... | lookup checkip subnet as client_ip OUTPUT subnet country city location latitude longitude | table subnet country city location latitude longitude rtt tx. ... View more

thanks for catching the error in the documentation. We will rectify this asap. ... View more

here are a few lines of example xml to create a map dashboard and to get latitude, longitude for ipaddress. (assuming you have events of sourcetype, access_combined_wcookie) MyMap sourcetype = access_combined_wcookie | iplocation clientip | geostats count by clientip ... View more

I am looking for the output after the lookup. that will tell what was the value of location, latitude, longitude that your lookup "checkip" returned. ... View more

so is your issue resolved now? Did you check to see whether the header line in the csv was compatible with your props.conf (I am guessing that was the issue). I am surprised that this error would not show up in splunk 5 system, if the props.conf stanza was getting used. ... View more

Hi, I get the problem that you are seeing, and understand what the csv lookup is doing. But if you can share the output upto the lookup command (which is fed to geostats), that will help understand the issue. You can try to obfuscate the non-essential fields. Also did you look into the search inspector, for warnings or errors? ... View more

What is the output of this command ... host="xxxxxxxx" | lookup checkip subnet as client_ip OUTPUT subnet country city location latitude longitude ... View more