| tstats dc(host) as hosts, dc(sourcetype) as sourcetypes where index="test" ... View more

What error are you seeing? What happens if you run the script manually? ... View more

For anyone who cares (or next time I run into this issue), ... I found and resolved my issue. I missed setting up a TCP Input on one of the 4 peer nodes. (This is why automation rocks, and doing stuff by hand is evil). I found that if I hit the /services/cluster/config endpoint on all the peer nodes, the one that was causing issues was returning ? for forwarderdata_rcv_port . Apparently the cluster master sends this bogus value to the UF's via the /services/indexer_discovery endpoint. Whoops. Find it in splunk like this: | rest services/cluster/config | search mode=slave forwarderdata_rcv_port="?" ... View more

Agreed in general. But I've hit a few scenarios where the error message is bogus (but only for the [GET /services/messages/restart_required/] command), all of the other commands seem to be CLI syntax errors as you point out. Here's an example: ~/cmaster/bin/splunk edit cluster-config -cluster_label "Lowell Idx Cluster" The cluster-config property has been edited. This command [GET /services/messages/restart_required/] needs splunkd to be up, and splunkd is down. I'll also point out that sometimes splunkd.log` show you the exact error, which makes me wonder why the CLI can't just report it? 😉 ~/cmaster/bin/splunk edit cluster-config -mode supermaster This command [POST /services/cluster/config/config] needs splunkd to be up, and splunkd is down. splunkd.log shows: 12-16-2015 19:20:04.516 +0000 ERROR ClusterStatusHandler - mode=supermaster invalid ... View more

I had this same error, but in my case it was caused by something weird with one of the indexers. I ran the CURL command noted above and saw that one of the 4 indexes was missing (or "replaced with hostport="?"). So I went to the "missing" indexer, ran a splunk offline and now the UF started working correctly and the CURL command returns only a valid list of indexers. Very weird. ... View more

Even stranger, I'm staring at a Splunk 6.1 outputs.conf that has an encrypted sslPassword in $SPLUNK_HOME/etc/apps/MY_Forwarder_Outputs/local/outputs.conf . So I'm going to disagree with the docs on this point. ... View more

Well this talks about SSL passwords, not pass4SymmKey under [indexer_discovery:] , but I suppose the same logic could apply. I'm a bit confused by this behavior. I understand why there are issues if you stick a password in a "default" directory, but I'm not sure what the issue is with updating entries in the "local" folder. ... View more

On at test I'm running, it looks like the encrypted version is stored under $SPLUNK_HOME/etc/system/local/outputs.conf instead of under the apps folder where the base setting is located. This seems even more problematic as system/local doesn't get updated. ... View more

Really great answer! Thanks for posting it. I love the use of install_name_tool over symlinking. I'm assuming this will allow me to run multiple versions of Splunk side-by-side (in different install folders, of course) which is something I need to do from time-to-time. ... View more

Subsearches will only return up to 10,000 results by default. Check out limits.conf docs for more details. You may want to consider rewriting the search to avoid using a subsearch by using the lookup command directly. ... View more

The docs say: Running a cluster across multiple sites is not currently supported. Search head clusters have been tested only with all members running on a single site. ... View more

I think the answer is no. Added an upvote because I'd like an authoritative answer to this as well. ... View more

So this question is old, Lucas not sure if you came up with a solution or not, but here's some additional info: First off, you don't want to simply replicate existing buckets as is. Duplicating your buckets by hand will result in Splunk seeing the data twice. Splunk has no way of knowing that the same bucket exists two places, so it will treat both (or all) copies as pre-clustered buckets and therefore search all copies. So not only do you have the overhead of searching the same data multiple times, but now you'll need some sort of "dedup" or other clever way to eliminate duplicates in your searches. Not fun. It's pretty easy to trick Splunk into converting non-clustered buckets into clustered ones. If you've taken a look at the bucket folder naming, you'll pretty quickly see the difference between the names of clustered and non-clustered buckets. The biggest difference is that the clustered buckets has the GUID as part of the name, which indicates which server the bucket originated. Keep in mind that the cluster master is essentially stateless between restarts, so everything it knows about the cluster is gleaned during the initialization phase; this mean that you can pretty easily trick Splunk into thinking that a non-clustered bucket is a single-site clustered bucket. (Multi-site clustering is a completely different beast in this regard. Splunk made it much more difficult to pull of this kind of a trick at a multi-site level.) Bucket conversion itself can be done using something like this: GUID=$(cat $SPLUNK_HOME/etc/instance.cfg | grep '^guid' | tr -d ' '| cut -d'=' -f 2) find $SPLUNK_DB -type d -regex '.*/db_[0-9]+_[0-9]+_[0-9]+' | ( while read bkt; do mv -v $bkt ${bkt}_${GUID} ; done; ) Use this at your own risk! Only run this on the indexes you want to replicate. Copy a small number of buckets over to a test server and do this all in a non-production cluster before you attempt this on prod... (And so on) Also keep in mind that there are additional conversation steps required beyond this, but this is the one bit in particular that's not really documented. Once your buckets are renamed, I suppose your could pre-replicate them out to a secondary node. However the bucket should be renamed from "db_" to "rb_" (indicating that it's a replicated bucket). Depending which version of Splunk you're running (5.x vs 6.x) where the replicated bucket should end up will be different. And of course if you already have multiple indexers, trying to pre-share this data gets a lot more complicated and probably isn't worth it. (In Splunk 6 the bucket replication is actually more efficient because it tries to copy both the raw and index data at once, whereas in 5.x Splunk would only copy the raw data and the require the indexes to be rebuild on the destination server; which consumed considerably more resources.) And again, if any of this seems difficult or confusing to you, and you value your data, please contact an expert. There's Splunk PS, and lots of Splunk partners who are qualified to help with this kind of conversion. There's also a number of pros/cons to consider with clustering in general, which is good to talk though with someone who's had experience maintaining a cluster. For full disclosure, I work for Splunk partner. ... View more

Here's an additional optimization that also solve the UI issue where users get the error: msg="A script exited abnormally" input="./bin/rlog.sh" stanza="default" status="exited with code 1" This is based on Splunk more recent releases of this script, but changes 2 things. It checks to see if the number of lines equals the seek value. So if there's no new lines to read, don't bother running "awk". It swaps out the super user check with a file readability check. This makes it possible to run as the "splunk" (privileged) user as long as you've setup file system permissions appropriately. if [ "x$KERNEL" = "xLinux" ] ; then #assertInvokerIsSuperuser test -r "$AUDIT_FILE" || exit 1 assertHaveCommand service assertHaveCommandGivenPath /sbin/ausearch if [ -n " service auditd status " -a "$?" -eq 0 ] ; then if [ -a $SEEK_FILE ] ; then SEEK= head -1 $SEEK_FILE else SEEK=1 echo "0" > $SEEK_FILE fi FILE_LINES= wc -l $AUDIT_FILE | cut -d " " -f 1 if [ $FILE_LINES -eq $SEEK ] ; then # No new events in audit.log exit 0 fi if [ $FILE_LINES -lt $SEEK ] ; then # audit file has wrapped SEEK=0 fi awk -v START=$SEEK -v OUTPUT=$SEEK_FILE 'NR>START { print } END { print NR > OUTPUT }' $AUDIT_FILE | tee $TEE_DEST | /sbin/ausearch -i 2>/dev/null | grep -v "^----" fi ... View more

First off, doing a "chmod +x" makes the scripts executable which explain why this started showing up. (If these files are not executable, then Splunk cannot run them. Most often this is a packaging bug.) So I just tracked down the error with rlog.sh . The following line is to blame: awk -v START=$SEEK -v OUTPUT=$SEEK_FILE 'NR>START { print } END { print NR > OUTPUT }' $AUDIT_FILE | tee $TEE_DEST | /sbin/ausearch -i 2>/dev/null | grep -v "^----" The problem is that grep will return a non-zero error message when it doesn't find what it's looking for. So in a normal case (on an active system), if new new events have been written to /var/log/audit/audit.log , then the line "---" will be written by ausearch and then grep will be happily remove it. (exit code 0). However, if there are no new events in audit.log then grep will be grumpy because it didn't have any work to do and exit with code 1. Since this is last line in the script to be executed, the script exits with grep's exit code, which is 1. Hence the error message you're seeing. So the reality is, we don't care about grep's happiness, so a real simple solution is to just always report success by explicitly returning 0. That can be done like so: awk -v START=$SEEK -v OUTPUT=$SEEK_FILE 'NR>START { print } END { print NR > OUTPUT }' $AUDIT_FILE | tee $TEE_DEST | /sbin/ausearch -i 2>/dev/null | grep -v "^----" exit 0 I'm not sure what's happening in sshdChecker.sh , and it's certainly conceivable that you're hitting a different issue with rlog.sh but you really just have to dig in to the shell script to see what's going on. Of course if you don't need the input, just disable it and the annoying error will go away. Update: A slightly more efficent option would be to not even execute the command unless new content was added to the audit.log file. This can be done by adding the following: if [ $FILE_LINES -eq $SEEK ] ; then # No new events in audit.log exit 0 fi ... View more