While early filtering is a good rule of thumb, in this instance remember the "where" command is categorized as a Distributable Streaming search process, so this would also be done at the index level and more importantly can be done BEFORE the final output, so it does not necessarily generate more traffic as Splunk will send it down as well knowing this fact about the "where" command. But, like I said, and learned from a great teacher I had, that is generally a good rule of thumb to follow 😉 Also, the above about Distributable Streaming goes for: eval, fields, rex, where, etc. For the curious, here's a great read to understand how searching works wrt different commands: https://docs.splunk.com/Documentation/Splunk/7.2.3/Search/Typesofcommands ... View more

thanks.. got it.. ... View more

Sorry to not post the resolution soon enough. we found that at one point we used Search head pooling. Then disabled it at later time and moved the Shared folder to a local location on the active server. During the upgrade this share folder was not upgraded. We upgraded it few days ago and alert emails started flowing. Thank you for the tips! ... View more

Thanks Ranjith. This worked. ... View more

Hi Antony, thanks for this, it sounds like we have the same problem. I haven't found any solution that gives me what I'm after. Have you? ... View more

Yes, both indexers are visible from both search heads. ... View more

For all your configuration, use local. Also please not that you might need to adjust the time prefix regex to match all your events ... View more

Best way is to set up development environment e.g. user's own workstation or in some development server if needed.. If you have valid splunk license and support contract you could ask development license from Splunk. Just fulfill this request: https://www.splunk.com/en_us/resources/personalized-dev-test-licenses.html ... View more

thanks.. will do the same... ... View more

Thank you so much 🙂 I managed to get it to work. All the best!! ... View more

You are welcome, Please mark as answer so that the thread will be closed ... View more

Out of interest, is the _cluster directory there for some special system reason ? We distribute indexes.conf and parsing apps for the indexers by putting them in apps under master-apps and we've never done anything with the files under _cluster. You recommend putting configs in _cluster (in here: http://docs.splunk.com/Documentation/Splunk/6.2.1/Indexer/Updatepeerconfigurations#Distribute_the_configuration_bundle), rather than appifying/containerizing all configs as per your normal best practice ? It strikes me as something you (Splunk) are likely to update as you release new versions and add indexes (for example, _telemetry). I'd call this a system directory, not a user manageable directory. Soooo...I was just wondering why it's there. ... View more

Hello Rakesh, Again I would do the adjustments in the search to get de-duplicated data. However, try below configuration. Please be aware that It's not recommended [my_index] coldPath = $SPLUNK_DB/my_index/colddb homePath = $SPLUNK_DB/my_index/db thawedPath = $SPLUNK_DB/cold/my_index/thaweddb maxHotSpanSecs =3600 --->rolls hot to warm after 1 hr maxWarmDBCount = 0 --> rolls warm to cold (since its 0,it will directly roll to cold) frozenTimePeriodInSecs = 60 -->cold to frozen rotatePeriodInSecs = 60 Please try this in a test environment and check the logs to see if it happens . Refer http://docs.splunk.com/Documentation/Splunk/6.1/Admin/indexesconf for variable explanations ... View more

remark, with a btool you will only find the public or shared saved searches from the system $SPLUNK_HOME/etc/system/(local|default)/savedsearches.conf and from the apps $SPLUNK_HOME/etc/apps/(appname)/(local|default)/savedsearches.conf If you want to find the "private" saved searched, you have to go in each profile from $SPLUNK_HOME/etc/users/(username)/(appname)/(local|default)/savedsearches.conf ... View more

Thanks for that tip. I don't need to delete anything because the data source was never actually working, hence why i wanted to remove it. ... View more

Thanks a lot, that made things clearer! I've used Cisco systems before, and I'm used to access lists where you read from the top down and use the first rule that applies to your current file. Of course this wouldn't make sense here, where the first default rule is an "accept all"-rule. ... View more

When I've had this issue, I chalked it up to entropy on the system over time. Assuming the config and the ports are all good, then I start with some simple and basic things: Run splunk stop and then do a kill -9 on the results of: ps -ef | grep -i splunk | grep -v grep | awk '{print $2;}' to stop anything that didn't actually shut down Restart the actual server thereby purging any zombie processes that might not be needed but are still running from usage over time When all else fails, reinstall that instance. This can be made easy through proper use of the configuration distribution features ... View more

Team, I haven't took backup of the system and upgraded to 6.4 on Linux OS whereby it has over written the config files, now I have lost everything related to config. Noticed that updated version is installed on a different path and Index remains intact, appreciate if you can suggest on how to proceed...planning to perform below Un-install Splunk – 6.4 Install older version 5.0 Try mapping Index’s that were existing Check configuration to see where applications are discovered ... View more

The only way is to get a license reset from Splunk. There are 2 ways to get this: 1: Open a support case. 2: Call your Splunk Salesman. Option #2 is usually way quicker than #1. If this is an emergency, DM me on slack. ... View more

Possibly too much information: I'm using these "top 10" tables with data bars for drilldown. I originally wanted to use tag clouds (which is what I'm using in equivalent dashboards on a different analytics platform), but it wasn't immediately obvious to me how to add drilldown behavior to either the custom tag cloud viz in the Splunk-built dashboard examples, or the Wordcloud viz. So I "fell back" to using tables (which support drilldown out-of-the-box via Simple XML), and added data bars to offer the feeling of "scale" that font sizes in tag clouds offer. In case you're wondering, I don't simply use bar charts with drilldown because, in my data, some values can be relatively huge, dwarfing the bar sizes of others (even in the "top 10"), so there's not much "bar" to click on for drilldown. This problem is avoided in tag clouds by having a minimum font size. ... View more

Normally it takes the scheduled time but Splunk considers different methods to run scheduled reports. http://docs.splunk.com/Documentation/Splunk/6.3.2/Report/Configurethepriorityofscheduledreports ... View more

Hi, But this is what it does... if you use this part: | rest splunk_server=* /services/shcluster/status |fields peers*label |transpose |table column "row 1" |rename "row 1" as label |append [ rest splunk_server=* /services/shcluster/status | fields peers*status |transpose |table column "row 1" |rename "row 1" as status] You will see that add the to tables in one. Then I extract the ID without .label and .status |rex field=column "[^\.]+.(?<id>[^\.]+)" And then with the stats I "join" both in one. |stats last(label) as label last(status) as status by id ... View more

You could use the Timewrap command: https://splunkbase.splunk.com/app/1645/ | timechart count span=1h | timewrap d | It allows you to compare days to days or weeks to weeks very easily ... View more

Please explain what is meant by ALL hosts? Do you have some hosts which are sending this data in without the correct timestamps? If so check that props.conf is on every indexer and forwarder that is sending the data in, even the universal forwarders. ... View more