I have recently added an independent search head which performs distributed searches using two different search peers, lets call them Peer A and Peer B. I want queries from the search head to only use Peer A by default, and be able to search Peer B on demand by specifying in the search terms (e.g. splunk_server=* or splunk_server=PeerB). The Splunk documentation mentions this is possible, but following the links just takes me in circles and doesn't ever explain how to do it. From Splunk 6.1.2 Documentation "When performing a distributed search from a search head, you can restrict your searches to specific search peers (also known as "indexer nodes") by default and in your saved and scheduled searches." I've read that I can specify default indexes by role, but since I have recently connected two indexers with this new search head, some of the index names are the same (e.g. "main") and so it seems to make more sense to set defaults for searches by splunk_server rather than by index name. How can I set the default search peer for distributed searches? Do I have to rename the indexes and then use the role rules to accomplish this? If so, are there any pitfalls or caveats to renaming the main index? ... View more

I'm on 6.1.2 -- I am troubleshooting some search quota issues and I think this is the root of the problem. I have some scheduled searches that are shared with other roles which run with an owner of "nobody", unless I change the permissions to private, in which case my user name is listed as the owner. I am getting this information from the scheduler log via this search: index=_internal sourcetype=scheduler | stats count by savedsearch_id Running the searches as "nobody" results in some skipped searches, as "nobody" observes the default srchJobsQuota = 3 in authorize.conf This means that in order to ensure the searches don't get skipped AND share the results of a regularly scheduled search, I need to either increase the default srchJobsQuota (which I don't want to do), or manually assign an owner with an appropriate quota to a search in savedsearches.conf every time I create a scheduled search with these requirements. Is this the intended behavior? Possibly related symptoms - I have also noticed that sometimes scheduled searches are automatically set with an expiration date as "saved" (even though I had not manually saved them or even viewed the search, I just let the scheduler start it, and observed it in the "Jobs" view) and other times they expire wildly early, not observing the default dispatch.ttl=2p (while other scheduled searches clearly do respect the setting). If this ttl behavior does not seem relevant, please ignore it and I will ask about it in another question. ... View more

My current Splunk Enterprise instance contains a lot of logged data, but would be exponentially more useful if I could easily correlate it with data that is currently housed in thousands (and growing) of individual databases. I'm looking for some very basic ideas on a solution - how can I get this meaningful data from the individual databases into Splunk, where it will then be at my fingertips? I've been told that Splunk DB Connect is not a reasonable solution because of the number of databases, but I don't have a great understanding of the limitations so I'm open to ideas. Any suggestions? ... View more

Im currently overhauling the search architecture and am looking to classify my data into types, some of which will have subtypes. Upon investigation, it seems that I could create automatic lookups to classify most of my types/subtypes, except for classifying based on when field!=value (I am unaware of a way to lookup something that doesn't exist). I could also use eventtypes to build all of the auto classification, which seem to be more flexible in terms of defining the data types (can use full search terms, not just matching key=value pairs). Is it more efficient performance-wise to use lookup tables over eventtypes? It seems that the lookup tables apply to only the sourcetype specified, while eventtypes would have to check every event regardless of sourcetype to see if there is a match. If I define an eventtype with some search terms and a sourcetype (e.g. sourcetype=thislog foo=value ), will Splunk only try to match assign that eventtype to the mentioned sourcetype? I read somewhere that there is at least SOME optimization for eventtypes, although it was not clearly documented or explained. Also, what about using a calculated field to add classification data? how does that compare to eventtypes or lookups. My searches typically return around 1 million events, with occasional searches returning maybe 300 million events. ... View more

I'm looking to create a timechart of counts for a field where there is one bucket per day, and each bucket spans back 30 days. So the search would return a count for -30d to now, and -31d to -1d, and -32d to -2d, and so on, ideally back to the oldest index entry. I intend to display the counts directly on a line chart and also do some basic evals on the counts (like percentages). The idea is to get a historical snapshot of data from the perspective of each day. I've read some other posts that point to summary indexing as an answer, and that seems like it would solve the problem for future data, but I also have a need to look at historical data right now rather than wait for the summary index to be built to be able to see a trend. ... View more