To rebalance data, splunk only offers you to rebalance one single index or all the indexes but there is no option to provide a list of them. I've created the following shell script to do this and I hope it will help the community. To use it: Create an empty directory and copy the script there Change the parameters at the beginning of the script to correspond to your platform Create a file named ~/.creds containing MY_CREDS="admin:password" (you can replace admin and password with any user account with admin privileges) Create a file in the same directory as the script named indexes.conf with a list of indexes you want to rebalance (one per line) Launch the script and provide a value for the timeout in minutes ("-m" switch;  e.g. "./rebalance.sh -m 2880") Tip: if you access your manager node using ssh, consider using "screen" to keep your session open for a long period of time.   #!/bin/sh HOST=my_manager_node.example.com PORT=8089 source $HOME/.creds CURL_OPTS="-su ${MY_CREDS} --write-out HTTPSTATUS:%{http_code}" INDEXES_FILE="indexes.conf" CONFIG_CMD="/usr/bin/curl ${CURL_OPTS} https://${HOST}:${PORT}/services/cluster/config/config -d rebalance_threshold=0.9" /cluster/manager/control/control/rebalance_buckets -d action=start" START_CMD="/usr/bin/curl ${CURL_OPTS} https://${HOST}:${PORT}/services/cluster/manager/control/control/rebalance_buckets -d action=start" configuration endpoint or by normal endpoint with action=start MAXRUNTIME_OPT="-d max_time_in_min=" INDEX_OPT="-d index=" STOP_CMD="/usr/bin/curl ${CURL_OPTS} https://${HOST}:${PORT}/services/cluster/manager/control/control/rebalance_buckets -d action=stop" STATUS_CMD="/usr/bin/curl ${CURL_OPTS} https://${HOST}:${PORT}/services/cluster/manager/control/control/rebalance_buckets -d action=status" LOG="rebalance.log" MAXRUNTIME="1" while getopts ":m:" opt; do case $opt in m) MAXRUNTIME=${OPTARG} ;; echo "Missing value for argument -$OPTARG" exit 1 ;; ?) echo "Unknown option -$OPTARG" exit 1 ;; esac done if [[ "$OPTIND" -ne "3" ]]; then echo "Please specify timeout in minute with -m" exit fi echo -n "Starting $0: " >>$LOG echo $(date) >>$LOG [[ -f $INDEXES_FILE ]] || exit echo "Configuring rebalancing" echo "Configuring rebalancing" >>$LOG EPOCH=$(date +"%s") MAX_EPOCH=$(( EPOCH + ( 60 * MAXRUNTIME ) )) echo "Will end at EPOCH: $MAX_EPOCH" >>$LOG HTTP_RESPONSE=$($CONFIG_CMD) HTTP_BODY=$(echo $HTTP_RESPONSE | sed -e 's/HTTPSTATUS\:.*//g') HTTP_STATUS=$(echo $HTTP_RESPONSE | tr -d '\n' | sed -e 's/.*HTTPSTATUS://') if [[ ! $HTTP_STATUS == "200" ]]; then echo "HTTP status: $HTTP_STATUS" echo "HTTP body: $HTTP_BODY" exit fi RUNNING=0 for INDEX in `cat $INDEXES_FILE`; do EPOCH=$(date +"%s") MINS_REMAINING=$(( ( MAX_EPOCH - EPOCH ) / 60 )) if [[ "$MINS_REMAINING" -le "0" ]]; then echo "Timout reached" echo "Timout reached" >>$LOG exit fi echo "Rebalancing $INDEX" echo "Rebalancing $INDEX" >>$LOG echo "Remaining time: $MINS_REMAINING" >>$LOG echo $(date) >>$LOG #HTTP_RESPONSE=$(${START_CMD} ${INDEX_OPT}${INDEX}) HTTP_RESPONSE=$(${START_CMD} ${INDEX_OPT}${INDEX} ${MAXRUNTIME_OPT}${MINS_REMAINING}) #HTTP_RESPONSE=200 HTTP_BODY=$(echo $HTTP_RESPONSE | sed -e 's/HTTPSTATUS\:.*//g') HTTP_STATUS=$(echo $HTTP_RESPONSE | tr -d '\n' | sed -e 's/.*HTTPSTATUS://') echo "HTTP status: $HTTP_STATUS" >>$LOG echo "HTTP body: $HTTP_BODY" >>$LOG if [[ ! $HTTP_STATUS == "200" ]]; then echo "HTTP status: $HTTP_STATUS" echo "HTTP body: $HTTP_BODY" exit fi RUNNING=1 WAS_RUNNING=0 sleep 1 while [[ $RUNNING == 1 ]]; do HTTP_RESPONSE=$($STATUS_CMD) HTTP_BODY=$(echo $HTTP_RESPONSE | sed -e 's/HTTPSTATUS\:.*//g') HTTP_STATUS=$(echo $HTTP_RESPONSE | tr -d '\n' | sed -e 's/.*HTTPSTATUS://') if [[ ! $HTTP_STATUS == "200" ]]; then echo "HTTP status: $HTTP_STATUS" echo "HTTP body: $HTTP_BODY" exit fi echo "$HTTP_BODY" | grep "Data rebalance is not running" >/dev/null if [ $? -eq 0 ]; then RUNNING=0 else WAS_RUNNING=1 RUNNING=1 echo -n "." sleep 1 fi done if [[ $WAS_RUNNING == 1 ]]; then # We need a CR after the last echo -n "." echo fi done   ... View more

It looks like this particular example (Table with Data Bars) does not work for me. Is there anything particular I should check ? Is it a bug ? I'm using Splunk 8.2.1 and the latest Dashboard Examples app (8.2.2) ... View more

Our company policy requires an SSL private key to be encrypted. Unfortunately, my script is using the requests library which doesn't support this, as written in its documentation (feature request : https://github.com/psf/requests/issues/1573) There seems to be a workaround by using a custom adapater (https://github.com/m-click/requests_pkcs12) which is using pkcs12 package files (which contain both the certificate and the key  and can be encrypted). I've installed this adapter under "lib" in my app directory (and I added the line of code to my script to append this path to the library path). But this adapter is requiring the pyopenssl library (shipped with Splunk, installed in the python default library path) but this one, in turns, requires ndg.httpsclient.ssl_peer_verification (not shipped with Splunk!). So the adapter fails to load. So in the end 2 questions: 1) Is anybody having to use client-side SSL authentication to query an API and is using an encrypted key ? How to make it work ? 2) Is there a clean way to add ndg.httpsclient.ssl_peer_verification to the libraries available to splunk so that pyopenssl can be loaded ? I've tried to add it to my app's "lib" directory but it seems that pyopenssl can not find it (maybe it expects to find it in the "default" directory ?) ... View more

I'm building an app where Splunk is receiving a large number of IDs and a property that I will need to sum over time.  Let's take for instance a simple example: _time id amount 00:00 A 1 00:01 C 8 01:01 B 2 01:02 A 4   At 01:03, a user asks "What is the sum of amount per id, for each id you've seen in the last 15 minutes" ? Take into account this is a very limited example, but you will have millions of unique ids and a big event rate (hundreds of events per second). The expected answer is: A 5 B 2   The first reflex is something like this: "stats sum(amount) earliest=0 by id [ |search id ]" over the last hour. But it's not really scable since, over time, the first part of the query will need to sum a lot of events. Then the second thought was to add an intermediary summary index which is doing the sum(amount) over a small period of time (15 min) and keep the result. Yes, it's accelarating but after a year or so, I will still have performance / scalability issues. In the end, then only thing I need is to keep the last value "sum(amount)" per id and continue counting based on this value every time you receive a new event. That's why I'm wondering if we could use the kvstore to keep counting, everytime we see a record, we simply update the records with : key = id value = value(id) + amount Anyone having a similar experience with KV Store or having a similar issue ? ... View more

Splunk documentation about metrics.log is nice but not entirely up to date and complete according to me. In section "Tcpout connections messages", we are missing the aggregation parameter for the values. Is the data reported (like _tcp_KBps) by "name", by "destIp" , by something else ... ? I've tried to figure this out by graphing _tcp_KBps over time and use different aggregation parameters but I have never been able to get a continuous line with a growing value (which should be the case). From time to time another field named "one_time_client=1" appears. What does it mean ? When UseACK is true, you get 2 additional metrics : max_ackq_size and current_ackq_size. They are not documented as far as I know, what do they mean ? ... View more