Cisco eStreamer for Splunk: How to configure inputs.conf to release monitor file when no more data is being added?

[sjaworski]

I am using the Cisco eStreamer for Splunk app to collect Sourcefire logs. What I noticed is that Splunk does not appear to stop monitoring files when data is no longer being added to the log file. The eStreamer client grows the log file to 10MB, then rotates to a new file name. Splunk just adds the new file to its list to monitor and never lets go of the old file. Eventually Splunk is monitoring thousands of files that are no longer collecting data.

estreamer.log.1447173668  estreamer.log.1447175714  estreamer.log.1447177772  estreamer.log.1447179767  estreamer.log.1447181889  estreamer.log.1447184029  estreamer.log.1447186139

[monitor://$SPLUNK_HOME/etc/apps/eStreamer/log]
disabled = 0
source = eStreamer
sourcetype = eStreamer
index = estreamer
crcSalt = <SOURCE>

How do I configure the inputs.conf to release the file after a certain amount of time or size? Or is there a better way?

[douglashurd]

A new Splunk Firepower solution is now available if you are using Firepower version 6.x. You can download the new eStreamer eNcore for Splunk and the separately installable dashboard from the two links below:

eStreamer eNcore
https://splunkbase.splunk.com/app/3662/

eNcore Dashboard
https://splunkbase.splunk.com/app/3663/

It is free to use and well documented but if you would like to purchase a TAC Support service so that you can obtain installation and configuration assistance and troubleshooting you can order the software from Cisco (support obligatory with this purchase). The Product Identifier is: FP-SPLUNK-SW-K9.

Regardless of whether you take up the support option or not, updated versions will be made available to all free of charge and posted on Splunkbase as well as Cisco Downloads.

[sjaworski · Accepted solution]

Here is the solution I came up with.
Since estreamer has to run on Linux, I used logrotate to rotate the files.

Create file estreamer in /etc/logrotate.d/
Add this configuration.

/opt/splunk/etc/apps/eStreamer/log/estreamer.log.* {
        missingok
        rotate 7
        daily
        compress
        postrotate
                /bin/kill -HUP `cat /opt/splunk/etc/apps/eStreamer/bin/estreamer_client.pid 2> /dev/null` 2> /dev/null || true
        endscript
}

To force a rotation immediately and troubleshoot any issues, execute logrotate -vf /etc/logrotate.d/estreamer

Next the inputs.conf needs to be adjusted.

Copy the inputs.conf file to the local directory of the eStreamer app if it does not exist.

Edit the inputs.conf file and instruct Splunk to blacklist the gz files created by logrotate.

[monitor://$SPLUNK_HOME/etc/apps/eStreamer/log]
disabled = 0
source = eStreamer
sourcetype = eStreamer
index = estreamer
crcSalt = <SOURCE>
#Add Blacklist not to collect GZ files
blacklist = estreamer.log.\d{10}.\d{1}.gz 

Verify Splunk is only monitoring the active estreamer files. Execute, /opt/splunk/bin/splunk list monitor | grep estreamer

[splunky@splkcollector local]# /opt/splunk/bin/splunk list monitor | grep estreamer
                /opt/splunk/etc/apps/eStreamer/log/estreamer.log.1448663187
                /opt/splunk/etc/apps/eStreamer/log/estreamer.log.1448663245

😛

[mikaelbje]

Hi!

Using your above solution but I end up with lots of copies of logs getting gzipped and suffixed, so it looks like it's gzipping what's already been gzipped and rotated, in a loop. Using the exact config you pasted.

Is your config still working?

[woodcock]

Even if you blacklist the rotated files, the forwarder will STILL have to sort through them and will slow down more and more and more. To avoid this, you need to setup another process to delete files that are very old (say 30 days or more), or move them somewhere. If this cannot be done (all files have to stay where the are, as they are), then you can do something like this:

https://answers.splunk.com/answers/309910/how-to-monitor-a-folder-for-newest-files-only-file.html

[sjaworski]

Thank you for the direction, the solution I came up with is below.

[MuS]

Hi sjaworski,

one thing that straight popped into my eyes is crcSalt = This in dangerous on rotated log files, because it could lead to the log file being re-indexed after it has rolled.

You could limit the monitored files by using whitelist and or blacklist or setup the monitor stanza just to watch this one log file

 [monitor://$SPLUNK_HOME/etc/apps/eStreamer/log/estreamer.log]

See the docs http://docs.splunk.com/Documentation/Splunk/6.3.1/Admin/Inputsconf for more details.

Hope this helps ...

cheers, MuS

[sjaworski]

Thank you for the direction, the solution I came up with is below.