Hi, I'm trying to compare stats from 2 different dates (sometimes not back to back) and I'm running into a wall because of subsearch limitations. I want to look at total count and average reqtime for grouped URLs for 2 different days and then find the difference between then. Because of the number of logs, I would like to limit the search to just those days if possible. index=f5 instance=test earliest=-4d@d latest=-3d@d|eval urlall=split(uri,"?")|eval url=mvindex(urlall,0)+"*"|stats count avg(reqtime) as avgtimeold by url|sort - avgtimeold|where count>100|head 30|rename count as countold| join url [search index=f5 instance=test earliest=-2d@d latest=-1d@d|eval urlall=split(uri,"?")|eval url=mvindex(urlall,0)+"*"|stats count avg(reqtime) as avgtimenew by url|sort - avgtimenew|where count>100|head 100|rename count as countnew]|eval avgtimediff=avgtimenew-avgtimeold|eval avgtimediffpercent=tostring(floor(avgtimediff*100/avgtimeold))+"%"|eval countdiff=countnew-countold|eval countdiffpercent=tostring(floor(countdiff*100/countold))+"%"|table url,countold,countnew,countdiff,countdiffpercent,avgtimeold,avgtimenew,avgtimediff,avgtimediffpercent The subsearch doesn't work because it takes longer than 60 seconds to return the results. I thought of using a lookup table, but the plan is to put this into a dashboard and I'm not sure how I could populate the lookup tables from there. If anyone has any suggestions on which way I could go, it would be greatly appreciated. TIA! ... View more

How does one handle the large CMDB lookup table (cmdb_ci_list_lookup.csv) that is generated in a large environment. My file reached 844M and caused sync issues as well as filling up the hard drive with old bundles. Any plans to switch this over to the KVStore? ... View more

I'm trying to add another search head to my search head cluster. I'm receiving the following error when I try and bootstrap it. [labsplunk-sh:/opt/sh2a/bin]$ ./splunk bootstrap shcluster-captain -servers_list "https://#.#.#.#:8088,https://#.#.#.#:8090,https://#.#.#.#:8091,https://#.#.#.#:8092,https://#.#.#.#:8093" In handler 'shclustermemberconsensus': CONFIGURATION ID MISMATCH server.conf [shclustering] disabled = 0 mgmt_uri = https://#.#.#.#:8093 pass4SymmKey = XXXXXXXX adhoc_searchhead = true conf_deploy_fetch_url = https://#.#.#.#:8088 It's a fresh Splunk tar, but the cluster is in use with deployed apps/configuration, so I don't want to delete everything and start over. The only thing I did to the SH was set the licenser, cluster-config, shcluster-config, and the deployer. This is the only line in the log to indicate the problem. Anyone else seen this issue or any suggestions on how to fix? I've restarted the whole cluster with no change. ... View more

I'm trying to set the timezone via a deployable app to my search head cluster. If I put the configuration in the etc/system/local, it works fine. If it's in the app directory, then it doesn't. I did have a user-prefs.conf in my etc/users directory, but it didn't have the TZ config present and I took it a step further and deleted it. Using find and grep, the file in apps is the only user-prefs.conf that has TZ in it. Can anyone shed light on why Splunk would ignore the configuration even though it shows up in btool? PS, this is a shcluster, so that's why the file is in default. TIA, Todd Works /opt/sh2a/etc/system/local/user-prefs.conf [general] /opt/sh2a/etc/system/local/user-prefs.conf tz = America/New_York /opt/sh2a/etc/apps/user-prefs/default/user-prefs.conf [general_default] /opt/sh2a/etc/apps/user-prefs/default/user-prefs.conf appOrder = search /opt/sh2a/etc/apps/user-prefs/default/user-prefs.conf default_namespace = $default /opt/sh2a/etc/apps/user-prefs/default/user-prefs.conf showWhatsNew = 1 Doesn't work /opt/sh2a/etc/apps/all_sh_base/default/user-prefs.conf [general] /opt/sh2a/etc/apps/all_sh_base/default/user-prefs.conf tz = America/New_York /opt/sh2a/etc/apps/user-prefs/default/user-prefs.conf [general_default] /opt/sh2a/etc/apps/user-prefs/default/user-prefs.conf appOrder = search /opt/sh2a/etc/apps/user-prefs/default/user-prefs.conf default_namespace = $default /opt/sh2a/etc/apps/user-prefs/default/user-prefs.conf showWhatsNew = 1 ... View more

Hi, I'm creating a multisite Splunk deployment with timezone differences. Since most users do not change their timezone perf and it's set to default, it could change depending on which search head and indexer they pull from. Is there a configuration setting that would set the timezone for the entire Splunk environment? I would imagine I would need to set the indexers and the search heads to ensure the results are static, correct? Any recommendations on how I should approach this? TIA, Todd ... View more

Hi, I'm trying to setup the Slack notification app and I'm having issues. When I use the webhook that I generated and CURL, i'm able to send messages to any channel. When I try and use the app with the webhook configured, I get the following errors: 07-12-2016 07:55:58.294 -0700 INFO sendmodalert - action=slack - Alert action script completed in duration=200 ms with exit code=0 07-12-2016 07:55:58.290 -0700 FATAL sendmodalert - action=slack STDERR - Sending the slack message failed 07-12-2016 07:55:58.289 -0700 ERROR sendmodalert - action=slack STDERR - Error sending message: HTTP Error 404: Not Found Can anyone give any guidance to how this is set up? TIA! ... View more