eStreamer Installation Tips for Distributed Splunk
I have a distributed environment with an Enterprise Security (ES) search head (SH), an ad-hoc search head and separate indexers. It wasn't too difficult to get eStreamer going but there are some confusing points that aren't well documented, so I'll share my experience and tips.
For this example, let say we'll will be running the collection script from an indexer, and we'll want to access the app on the ad-hoc SH as well as the ES SH.
First off, the estreamer for Splunk app contains a perl script which is how it collects data. I read that it needed to be installed on an indexer, but now that I see how it works, I think you could install it anywhere (SH or heavy forwarder), as long as you are forwarding your SH events to your indexers.
At the end of this, you will have installed 'eStreamer for Splunk' on the indexers, ad-hoc SH and ES SH (you only need it on one indexer, but this will ensure the estramer index is created on both), and 'Splunk Add-on for Cisco FireSIGHT' on the ES SH.
Indexer:
You'll need to get the perl dependencies installed (in this case on the indexer), and there is one that CAN cause you a headache if you use CPAN. Use the OS package manager for the SSL packages: yum install openssl-devel perl-Net-SSLeay
Install these first themn the rest of the perl modules should install fine with CPAN.
Install the eStreamer for Splunk app. To test, run the bin/estreamer_client.pl and make sure there are no errors (you should see usage instructions when it runs).
On the indexer, go to the eStreamer app from the web interface to set it up. (There's also a 'config_nogui.sh' if you have disabled web or are otherwise inclined.) It requires the IP, port and full cert path (this should be provided to you by the sourcefire admin). Once these values are set, uncheck the "disabled" check box and click 'Save'.
Hopefully at this point you start to see events on the estreamer status dashboard (we're still on the indexer at thsi point, but this is just for verification). The logs are collected in the eStreamer app directory under log/.
Ad-hoc SH:
Now, you obviously don't want people to access the indexer for the app, so you go to the ad-hoc SH and install the eStreamer app. I don't think you need to install any of the perl dependencies here, as we will not be using those scripts on the SH. When you access the eStreamer app you'll see that it wants you to setup the collection script in order to see the app.
To get around this, create the app.conf file under local/ and add these lines:
[install]
state = enabled
is_configured = 1
It should now bypass the config screen here on your SH.
Now you will see that on the eStreamer Status dashboard the client says it's in an error state. This is because you are not running the client on the SH and this dashboard is picking up the status of all of this scripts (client_check.py) output in your environment. To get around this, just change the dashboard panel to only include the host that you have your collection script running on (in this case, your indexer). (Add 'host=' just before the '|eval ..' in the search string.) More thoroughly, you should disable the script input $SPLUNK_HOME/etc/apps/eStreamer/bin/client_check.py on the SH and any other place you are using the app but not running the client.
ES SH:
Now, if you have Enterprise Security (ES) running, there's another app that helps with all of the eventtyping and tagging needed for ES to bring these events in, Splunk Add-on for Cisco FireSIGHT. You need to install this on the ES SH. The issue now is that the sourcetypes may not be in sync between this app and the eStreamer app. It depends on the version of Sourcefire you have. In my case it required the FireSIGHT app to see cisco:sourcefire as the sourcetype, so I changed it (local/inputs.conf on the indexer). But then the eStreamer app stopped working because the sourcetype in that app is set to look for sourcetype=estreamer. So again, it depends on your version, but for me I moved over all of the conf files that contained sourcetype=estreamer to the local directory, and changed it to my new sourcetype. I did it on both the indexer and the SH but I believe these are only search time configs so you should only need to do this on the SH (anywhere you run the eStreamer app).
Then everything worked.
I hope this helps others get by some of the obstacles for this setup. It's not the most straightforward process in a distributed environment, nor when the collection script is tied to the app this way this is.
... View more