Splunk ES on SHC's is not supported on Windows but is supported on Linux. http://docs.splunk.com/Documentation/ES/5.0.0/Install/DeploymentPlanning#Splunk_Enterprise_Security_and_search_head_clustering ... View more

Can't you simply disable the KVStore on the indexers via server.conf? [kvstore] disabled = true|false * Set to true to disable the KV Store process on the current server. To completely disable KV Store in a deployment with search head clustering or search head pooling, you must also disable KV Store on each individual server. * Defaults to false. ... View more

It's hard to do without an actual sample and what it would look like but perhaps this would help? sourcetype="bn22_epsin" earliest=-1h latest=now | dedup Cpk_Num | stats count by Cpk_Num | where count=0 ... View more

According to the known issues list, this particular known issue relates to event routing to multiple targets. I'd recommend opening a case with Splunk Enterprise Support if you think you're being affected by this issue. If not, just post whatever your problem is and perhaps we can assist. ... View more

That did the trick for me. ... View more

By installing the Splunk app for *nix, and deploying the *nix TA to your Linux servers, you can have your servers report the output of the "df" command in Splunk. You can then do a search on your collected "df" data to find servers that have 85% or higher disk utilisation. For this you would need: A Splunk server (search head / indexer) A Splunk Deployment server to push out the *nix TA (Doesn't need to be a separate server) A Splunk universal forwarder installed on your 5000 Linux servers that is connected to your deployment server. Do you have any of this in place yet? Secondly, are you sure Splunk is the best answer for your problem(s)? You don't have any performance metrics collection in place already? ... View more

The only thing I can find that would help in some way is the following from transforms.conf: batch_index_query = <bool> * For large file based lookups, this determines whether queries can be grouped to improve search performance. * Default is unspecified here, but defaults to true (at global level in limits.conf) allow_caching = <bool> * Allow output from lookup scripts to be cached * Default is true The first one only works for file-based lookups and since you're using an external script, it won't help you much and the second one doesn't do anything for new requests. You'll either have to implement the logic in the external lookup script itself or change your approach altogether. How often do people's phone number change? Can you Request the whole list and use that as a file-based lookup in Splunk? Perhaps updating the full list once a day? Even with a million-line CSV file as a file-based lookup, Splunk will still enrich the data charmingly. ... View more

Use multikv to separate the data and then use the new values for your chart/timechart. index=... sourcetype=... | multikv fields PollTime, "Server Name", QueueName, "Display Name", value | timechart avg(value) by "Server Name" You might have to tweak the above query a bit but that should get you started. ... View more

You can't. See here for all the options you can do via the CLI: http://docs.splunk.com/Documentation/Splunk/6.2.5/Data/MonitorfilesanddirectoriesusingtheCLI Adding a blacklist isn't one of them. You could probably do it with some regex/sed-magic. You could search for you particular monitoring stanza and add the blacklist line underneath it and then restart splunk. This should help: http://stackoverflow.com/questions/15559359/insert-line-after-first-match-using-sed ... View more

Which exactly what I'm saying. Either use the Splunk start/restart script in /etc/init.d/ and hook in your python script there. This will run your script when you start/restart Splunk. Or Create a scripted input in Splunk and give it an interval of -1 so that it starts every time you start/restart Splunk. I recommend this option. See the documentation link I provided earlier. ... View more

not /opt/splunk/etc/init.d/ but /etc/init.d/ of the actual server. ... View more

If you're running Splunk on UNIX/Linux, you could edit the startup scripts in /etc/init.d/ to include your script in the start/restart section. Just be careful that if you ever do a "splunk enable boot-start", your changes will get overwritten. You can also create a scripted input and set the interval to -1 as per the documentation : http://docs.splunk.com/Documentation/Splunk/6.2.5/Data/Setupcustominputs For one-shot data streams, enter -1. Setting interval to -1 will cause the script to run each time the splunk daemon restarts. ... View more

http://answers.splunk.com/answers/439/how-to-convert-second-to-hh-mm-ss-format-in-the-exported-search-result.html Bit of googling found that. ... View more

Easiest would be | rex field=_raw ".*duration\s\[(?<duration>.*?)\]" This will capture everything in between the [] after the word "duration". So in your examples, it would be "1.1s", "999ms", and "814ms". Is that what you're looking for? ... View more

| rex "Employers Name=(?<employersname>[^,]*)" | rex "Providers Name=(?<providername>[^,]*)" | eval contactname=coalesce(employersname,providername) | table contactname This should extract both the Employer's Name (if it exists) and the Provider's Name (if it exists) and fill the field "contactname" with the employers name, unless that's empty, then it'll put the providername in there. I see from your logging that it's all key/value pairs, did you try using the | extract pairdelim=",", kvdelim="=" keyword to automatically extract these fields? Splunk will extract the Employer's Name and Provider's Name if they exist and that should solve all your problems. ... View more

do you have some sample logging that you could post here (just a few lines)? ... View more

| rex field=_raw .*cs1=(?[^\s]+)\scs1Lable=(?[^"]+) There's a slightly better version of the regex. This one doesn't capture the closing "-symbol in your label field. As for your second question, have a look at http://answers.splunk.com/answers/78340/is-it-possible-to-set-field-name-and-value-with-rex-similar-to-1-2-in-transforms-conf.html ... View more

| rex field=_raw "####<[^>]+>\s<(?[^>]+)>\s<[^>]+>\s<(?[^>]+)>\s<(?[^>]+)>\s(?.*)" That'll do the extracting you want but like the other said, you'll need to fix your linebreaking and timestamping via your props.conf. ... View more