I'm looking at several different options for having our central Splunk server trigger a behavior on our forwarders when certain events occur. And I'm not a developer...
I'm guessing the "right" way to do it is have a script that will take data from the search pipeline and run a script that accesses the REST API on the remote host.
The other way I can think of doing it is to have the central Splunk server use a script to send a syslog message to the forwarder. A saved search on the forwarder would process the syslog events and trigger the appropriate script.
Is it possible to configure a heavy forwarder so that it only indexes data from a single source or sourcetype and it just forwards everything else?
It is definitely possible with heavy forwarder to index locally and same time do the forwarding as well. And splunk do have documentation available for your requirement. Please refer to Splunk documentation for Routing and filter data and look for section "Index one input locally and then forward the remaining inputs" for details specific to your requirement. This is something I will be working tomorrow 🙂
If you have a full install of Splunk, not just the forwarder, yes. You configure inputs.conf to index only certain things and use outputs.conf to send the rest off to another system.
If you want to control the remote system, it is likely better to use the Deployment Server and have a script triggered that alters the DS config setup for the remote system. Then have that system call into the DS to see it's config changed.
However, it is usually better to simply have a forwarder forward data and an indexer run the searches that cause things to happen. You can segment data in an indexer fairly well if you use separate indexes for different data sources or systems that require different people to access the data.
You can send all events to the central/primary indexers from the forwarders. Then you have the indexer run a script that can use whatever network protocols you have available (e.g. ssh w/ pubkeys on *nix or net/wmi/powershell calls on Windows) to tickle actions on the remote systems. I have all scripts run on the search heads and/or indexers, even when they have to cause action on a remote system. The delay is only moments based on network latency and the brief indexing time cost as the data comes across the wire, hits the indexer, gets indexed, and then triggers the real-time search.
I wasn't being clear. It's not that I'm trying to control the config of Splunk, I'm trying to make forwarders able to "react" to various events by running a search/script. I'd rather keep the indexing load centralized, so I want to route syslog to the indexQueue and then all of my other inputs should just be forwarded, not indexed.