won't you need to cater for the stuff either side (spaces/dashes? rex "cmdbRequest\s(?<Request_ID>\d+)\s\-\sTransaction" ...or something like that, depending on how consistent you expect it to all be. I've not tested it, but it might put you on the correct path ... View more

Have you looked in rpc.log and - more likely - dbx2.log as well? I assume this happens after you've configured it, so what does the dbx app config look like (inputs.conf) ... View more

Thanks for the response. Even when I have N number of indexers it doesn't work (even though the cluster is green and indexing), but I tried your suggestion anyway - still doesn't work however. [edit]: Hmm, perhaps I stand corrected - the events no longer seem to be occurring in the logs, but I still can't wget that URL (and I can when clustering is disabled on peers), so I'm guessing the master has just stopped trying. This is possibly just normal? It's the only operational cluster I have access to at the moment, and it's the simplest one imaginable, with a complete rebuild producing the same result. ... View more

I believe the OP is talking about an index cluster, not a search head cluster - and I have the exact same problem (latest 6.3). When I comment out the clustering section, which on a peer only has "master_uri=" and "mode = slave", the splunk instance starts and runs, but as soon as I add them back in, both 8000 and 8089 become unavailable. The cluster master seems to report that peer being added correctly, but it reports this in the logs constantly: 10-19-2015 02:26:22.870 -0700 ERROR HttpClientRequest - HTTP client error: Read Timeout (while accessing https://192.168.141.130:8089/services/server/info) 10-19-2015 02:26:22.871 -0700 ERROR HttpClientRequest - HTTP client error: Connection reset by peer (while accessing http://192.168.141.130:8089/services/server/info) 10-19-2015 02:26:22.871 -0700 WARN DistributedPeerManagerHeartbeat - Unable to get server info from peer: http://192.168.141.130:8089 due to: Connection reset by peer ... View more

I'm using Ansible, not Chef, but the concept should be the same - and I'm currently dealing with the exact same problem myself. What seems to be working for me so far is: Spin up a Splunk instance somewhere that sets all of your pass4SymmKey stuff to the encrypted version of what you want Take the resulting encrypted password and use it in your recipes/templates/whatever Add $SPLUNK_HOME/etc/auth/splunk.secret to the list of files you manage with Chef, so that all instances of Splunk use the same secret to create passwords. Now when splunk starts up and checks to see if it can find an encrypted password, it'll pass I've not tested this for everything yet, but my local dev environment that I'm using to build my Ansible playbooks has an operational cluster using this method - I've only actually tested the theory on the cluster key in the last few minutes, but will get on to the rest of it. You'll obviously want to protect this secret - I use ansible-vault, but I guess encrypted data bags are probably your version. It's very early days in my testing, but I can update here with more detail if people are interested - or somebody can update me with the wall they know I'm going to hit but haven't yet 🙂 Cheers. ... View more

I have the same problem here. I've created/configured the app on one search head, and this doesn't happen, but when I deploy that app to a search head cluster, I get the error. It only seems to affect dbx logging though, so while it's possibly stopping me from debugging my problem where DBX doesn't work in the cluster, I don't think it's actually much of a problem. 🙂 ... View more