Splunk doesn't have any built-in features to send messages to AMQP. There is a AMQP modular input, but that takes messages from AMQP and indexes them into Splunk, not the other way around. https://splunkbase.splunk.com/apps/#/page/1/search/amqp/order/relevance Since there isn't an option in Splunkbase, you'll need to create something yourself. In 6.3 Splunk added custom alert actions, which let you integrate your own alert responses into Splunk. Yes you'd need to write a script, and the integrate that into Splunk. Take a look at the links below. There are many examples of other types of alert actions in Splunkbase that you can use to get started. http://docs.splunk.com/Documentation/Splunk/6.3.3/AdvancedDev/ModAlertsIntro https://splunkbase.splunk.com/apps/#/order/relevance/search/alert%2520action ... View more

So, lets break down what is happening when you schedule and run a summary search. When you run the search, like any search in Splunk, your search is distributed to the search peers (indexers) configured on the search head. The search head then takes the results of the search and stores them in stash files that it puts into $SPLUNK_HOME/var/spool/splunk directory. By default, Splunk has setup a batch input for this directory. So when the files are dropped into the spool directory, Splunk indexes the files and deletes them. If the search head is configured to forward its data, then it will not index the files locally, but instead forward them to wherever its forwarding destination is. What that means is that the source of the summary search results is not connected to the summary indexing destination. Now in practice, most everyone has their search heads configured to send data to their indexers, the same indexers they have configured as search peers. That's a best practice. So lets look at the scenarios you mentioned: If you run the summary index on a search head, and that search head is configured to forward data to your indexers, then any summary data will be evenly distributed among the indexers. This is what you want to do. There isn't any need to think about "distributing" the searches to the indexers, or distributing the results across the indexers, the search head takes care of that for you. If you run the summary index on a search head, and that search head is not configured to forward data, the summary results will be indexed locally on the search head. You might want to do this, but then you'll have to deal with storage on the search head, and as the summary result set increases in size, your search won't scale accordingly-- you'll lose the benefit of distributed search. If you run the summary index on an indexer, the data will remain on that indexer. You don't want to do this, because you have multiple indexers, so your search results will be incomplete. In general, you don't want to execute any searches directly on your indexer. Let the search head distribute them. ... View more

The most straightforward way to accomplish this search would be to find your matching events and perform your lookup. For example, lets say your sourcetype is vpn and your username is in the user field: sourcetype=vpn | lookup your_lookup user OUPUT location, bu, etc. | stats count by location, bu, etc. Since you have 150 fields, you should list out just the fields you actually will work with in your OUTPUT statement, rather than pulling the entire set. This search works well if you want to report on all, or nearly all of the VPN events you have. A slight optimization would be to do the lookup AFTER some consolidation: sourcetype=vpn | stats count by user | lookup your_lookup user OUPUT location, bu, etc. | stats sum(count) by location, bu, etc. Now, if you have say, thousands of users with activity, but only want to report on a small subset, you could start the search with a subsearch to narrow down your matching events. This search would only find users whose location is "USA". sourcetype=vpn [ | inputlookup your_lookup | search location="USA" | fields user] | .... However, given that your result set is only maybe 3-5k rows, performing the subsearch probably won't get you much. And, you run the risk of hitting a limit if the subsearch returns too many field values (ie, if more than 50k users are in USA). If I have a search that returns millions of rows, and I know I really only want a small subset of those based on a value from my lookup data, I might use a subsearch to narrow the list down. Otherwise, I'd perform the first search. Its much simpler. ... View more

It sounds like you are dealing with a single forwarder and a single inputs.conf file. However, it doesn't hurt to check the following: 1) another inputs.conf on the same host that is still configured to send the sourcetype. You can use btool to check and see what is monitored on your host: splunk cmd btool inputs list 2) Other forwarders sending the same sourcetype as well. Even if the host field matches, you could potentially have another forwarder with the same host=hostname stanza in the inputs.conf. I've seen this a few times when people have copied their forwarder installs or inputs.conf files around. If neither of those options pan out, assuming you modified the right file and restarted the forwarder, you might be seeing events that were timestamped in the future (or had misinterpreted timestamps that were indexed with a future date). This would result in you seeing data that looks "current" but was actually indexed in the past. Try this search: sourcetype=log4j | eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S") | table _time, indextime, _raw Do _time and indextime match? ... View more