Encore / eStreamer Client 4.8.1 Hex Flood or Failure to Ingest

[Pcktech]

Add-on: https://splunkbase.splunk.com/app/3662/

Known Affected: 4.8.1

Symptoms:

You begin to predominantly see Hexadecimal events in your Cisco FireSIGHT Index/Sourcetype instead of real data, and you see large gaps between events (usually ~10 minutes, the time it takes for it to roll over a file). The 'Source' also ends with '.log.swp' instead of '.log'.

Cause:

$SPLUNK_HOME/etc/apps/TA-eStreamer/default/inputs.conf 

[monitor://$SPLUNK_HOME/etc/apps/TA-eStreamer/bin/encore/data]
disabled = 0
source = encore
sourcetype = cisco:estreamer:data
crcSalt = <SOURCE>

The issue I believe is with the bolded line 'source = encore' because 'crcSalt = <SOURCE>' is also specified. Since all files have the same Source, all files have the same crcSalt which is why the actual '.log' is not collected. The '.swp' manages to get collected as Splunk checks the '.log' and since swp is a very short lived file Splunk accidentally collects a lot of garbage unrelated to the actual file contents (sorry Linux Admins for butchering the technical detail).

Solution:

Edit $SPLUNK_HOME/etc/apps/TA-eStreamer/default/inputs.conf and comment out the Source line, then restart Splunk services.

If someone knows of a way to override (via Local inputs.conf) source back with the filename (which changes frequently) so editing a Default inputs.conf is not necessary, please comment below. Those with the Cisco license allowing TAC Support on this add-on may want to raise this issue with them so they can fix it for new downloads and future versions -- I lack that particular license.

Hope this helps someone (I did a search for encore and hex and didn't see any prior conversation on the topic).

[anshu_splunk]

I'm not sure if the crcSalt is necessarily the problem.  I would try setting a "whitelist" attribute for the inputs stanza to only allow for files ending in ".log" to be monitored.  It takes a regular expression, so something like 

whitelist = \.log$

This would filter out the log.swp files.  Also this method would preserve the stanza header.  You can create this stanza in local/inputs.conf and just set the whitelist attribute there, while still being able to upgrade the add-on later if there's any changes to default/inputs.conf.  

[Pcktech]

Looks like you're right. It wasn't the issue. I added your whitelist suggestion to help keep the hex out when/if the issue ever occurs again. It didn't help fix the problem, but it kept the garbage out.

I also followed the _internal suggestion of increasing the inputs.conf initCrcLength value for the stanza from its default of 256 bytes to 1024 bytes (nice round number). This seemed to help restore data collection after the issue returned today following a restart of Splunk services (and subsequent restarts again didn't help). Hopefully this'll help prevent the issue from recurring.