2 searches to illustrate: | noop | stats count | append [ savedsearch my_accel_search ] | savedsearch my_accel_search | append [ noop | stats count ] The second runs in less than 20% of the time. Also of interest, the 2nd case has lots of notices about using summary search info in the job inspector. The first does not. No, this isn't my actual use case. But it illustrates the problem I'm having using a saved search in a dashboard. It makes the acceleration useless because it takes so long to run. Edit: The report acceleration summaries page does not increment the count in the case of search 1. Search 2 does. Based on the evidence, I'm thinking acceleration doesn't play with subsearches, joins or appends. Expected? ... View more

We have different groups in our company. Each group gets an app, say acme_search. There are roles associated with the app, for example acme_user. This role is then mapped to an LDAP group. One of the template files included with the base app install is acme_search/default/user-prefs.conf which contains: [role_acme_user] default_namespace = acme_search I've noticed just recently that all users are being sent to the search app by default. I swear this used to work as expected. This really sucks because newbs start creating artifacts, reports, alerts, dashboards, etc in search instead of their "Acme" app. And untangling that mess is not trivial. Not to mention the confusion it sows in the user base. It appears that the (relatively new?) default app user-prefs is overriding my per app config file. In a standalone SH env, if I put that stanza into user-prefs/local/user-prefs.conf it works. Doing that in an SHC env is tricky/hacky/gross. I presume it would work in system/local as well, while also presenting SHC problems. More to the point, I need it to be bundled with the app. Trying to manage a common user-prefs.conf file for 400+ different roles is not something I want to tackle. FWIW, btool shows good, but this setting has no impact on actual use. > splunk btool user-prefs list role_acme [role_acme_user] default_namespace = acme_search Am I missing something? Any workarounds? Linux Splunk 6.5.1 (tested on 6.5.2 as well) SHC and indexer cluster Thanks, jon ... View more

My first few attempts at rebalancing were pretty great. No muss, no fuss. They ran for about 12 hours and like magic my cluster was firing on all cylinders. Beautiful. 'Stuff happens' and I'm now in the situation again where I've introduced new servers to the the cluster (replacing old). Now I'm way out of balance. "No problem," says I. "Data rebalancing is awesome." Not so fast. Literally. I fired it off late Friday night. By Monday morning the process was reading 0.14% done, .01% more than right after starting the process 56 hours previously. By my math that's about 650 days to complete. I stopped the process, and restarted it for 1 index only -- 648 buckets using 1 TB of disk. After running for the last 18 hours, it's at 3% complete. So slow as to not really be usable. 12 servers in the cluster, 4 are new; all are: 20 core Xeons (2 threads each) 22 1.6TB SSDs 128 GB of RAM Splunk 6.5.1 Avg load over the weekend was under 2 Any suggestions appreciated. EDIT: > splunk btool server list clustering | grep max_peer max_peer_build_load = 2 max_peer_rep_load = 5 max_peer_sum_rep_load = 5 Jon ... View more

Inputs stanza from btool: [monitor:///apps/Logs/*/www/Reporting/CRTLog.log*] _rcvbuf = 1572864 disabled = 0 host = apphost1 index = reporting_main sourcetype = reporting_crtlog The log rotation they use keeps 10 rolled copies, named with .1-10 on the end. Eg, when the original rolls it gets named CRTLog.log.1 and a new CRTLog.log file is created. Standard stuff. I have confirmed, without a doubt, the rolled files maintain consistent content. I wrote a script to grab checksums of the first 1KB of each file every few seconds. They always check out -- .1's checksum matches what the original showed before rolling. However, Splunk is sometimes (not all the time) treating the 1st rolled file as a new file: WatchedFile - Will begin reading at offset=0 for file='/apps/Logs/apphost1/www/Reporting/CRTLog.log.1' Probably 30% of the time it re-reads the rolled file. Only .1, never any of the others. Any tips to further troubleshoot this? (Ticket's open, but after 3 days I kinda need an answer.) EDIT: Sample checksum comparo: I use for f in $(ls); do echo -n "$f: "; head -50 $f | md5sum; done to grab a list: CRTLog.log: 0fb375c11ad382eec3cc482fb1332c81 - CRTLog.log.1: 40f3878392f5ca816bfc4948b263d0e2 - CRTLog.log.10: ffc1a6dec71a64f69a2f4c42b53d68cb - CRTLog.log.2: a3b7d786d8aa7260cc5e46635e764c8f - <snip> Then wait for a roll to fire and grab the new list: CRTLog.log: ad978fdb89b04169e95ba96c15887042 - CRTLog.log.1: 0fb375c11ad382eec3cc482fb1332c81 - CRTLog.log.10: 82d1b645c89e4e34b4e0a89712d30f3e - CRTLog.log.2: 40f3878392f5ca816bfc4948b263d0e2 - CRTLog.log.3: a3b7d786d8aa7260cc5e46635e764c8f - <snip> So the first 50 lines (about 16 KB worth of data), matches before and after roll to .1. Splunk re-read the file in this case. ... View more

We're seeing about 50 ms. Normal range? How would we optimize this? EDIT: Also, how do you measure HEC performance from Splunk's point of view? I don't see anything in metrics that explicitly calls out HEC. Since the end user can adjust source arbitrarily, how do you isolate HEC stats from metrics? ... View more