Cisco eStreamer for Splunk: How to configure eStream flows retention?

[DomenicoFumarol]

Hi everybody,
we have the Cisco eStreamer for Splunk in our environment with the flows enabled that unfortunately are filling the disk space available very quickly.
Can we remove, via job script, older (more than a week for instance) estream flows or do we need to store them?

We can't find any field on the eStreamer App setting page related to flow retention.

Thank you.

[douglashurd]

What disk space is being filled?

The Flow data is an all or nothing proposition unless you use the Correlation engine to send you flow data when the flow meets some sort of criteria.

[douglashurd]

According to one of the experts here the best and easiest way to fix would be to create a symlink from one directory to the other :

ln -s /opt/splunk/splunk/etc/logs /splunkfs/estreamer/logs

What the command above is doing is symbolically linking the first directory to the second so the system will think its writing to the former but really writing to the desired latter.

[DomenicoFumarol]

Thank you for your reply, but I think the symlink will only store the log files in another location that will have a maximum capacity too.
My question was instead around the need of having these estream flows or perhaps we can remove them, without losing useful data, after a period of time (i.e. a month).

If we setup a cronjob that will remove these estream flows older than 2 weeks, are we losing the capability of searching events older than 2 weeks or perhaps those are saved somewhere else?

[afamuyiwa]

I'm running into the same issue as well. I have estreamer setup on a heavy forwarder with limited storage space. Does anyone have recommendation how much disc storage is required for the estreamer.log for at least 7 days of retention to start off with?

[DomenicoFumarol]

The path $SPLUNK_HOME/etc/apps/eStreamer/log on the indexer, configured as client eStreamer in FireSIGHT, containing logs in the form estreamer.log.1481198246.