All Apps and Add-ons

Estreamer failing after 6.2.01 upgrade

panovattack
Communicator

Recently updated FMC to 6.2.0.1. Estreamer client now only sends 5 or so events and then the estreamer client fails, both on Splunk and host-based client testing. Also, the server does not seem to respond to changes in the event type delivery options. Is the estreamer APP not compatible with FMC 6.2.0.1?

0 Karma
1 Solution

douglashurd
Builder

A new Splunk Firepower solution is now available if you are using Firepower version 6.x. You can download the new eStreamer eNcore for Splunk and the separately installable dashboard from the two links below:

eStreamer eNcore
https://splunkbase.splunk.com/app/3662/

eNcore Dashboard
https://splunkbase.splunk.com/app/3663/

It is free to use and well documented but if you would like to purchase a TAC Support service so that you can obtain installation and configuration assistance and troubleshooting you can order the software from Cisco (support obligatory with this purchase). The Product Identifier is: FP-SPLUNK-SW-K9.

Regardless of whether you take up the support option or not, updated versions will be made available to all free of charge and posted on Splunkbase as well as Cisco Downloads.

View solution in original post

0 Karma

douglashurd
Builder

A new Splunk Firepower solution is now available if you are using Firepower version 6.x. You can download the new eStreamer eNcore for Splunk and the separately installable dashboard from the two links below:

eStreamer eNcore
https://splunkbase.splunk.com/app/3662/

eNcore Dashboard
https://splunkbase.splunk.com/app/3663/

It is free to use and well documented but if you would like to purchase a TAC Support service so that you can obtain installation and configuration assistance and troubleshooting you can order the software from Cisco (support obligatory with this purchase). The Product Identifier is: FP-SPLUNK-SW-K9.

Regardless of whether you take up the support option or not, updated versions will be made available to all free of charge and posted on Splunkbase as well as Cisco Downloads.

0 Karma

panovattack
Communicator

eNcore + 6.2.2 solved our issue. However we still struggle with finding the multi-processor settings for eNcore. eNcore runs as a single thread, not sure how to make it multi processor. we configured our HF just to do this.

0 Karma

BHeindel
New Member

There is a hotfix available for FMC code 6.2.0.1 that we applied, that makes streamer function MUCH better.

I had to open a TAC case to get the hotfix for 6.2.0.1 - (bug CSCve44987)

I am told the same hotfix is rolled up into update 6.2.0.2 if you prefer to go that route.

0 Karma

ChrisBell04
Communicator

@BHeindel@idahopower.com
confirmed that 6.2.0.2 (build 51) fixes a lot of the estreamer issues present in 6.2.0.1.

0 Karma

jkleensang
Path Finder

I have the same issue, tho I can't confirm it was due to an upgrade... still waiting for word back from that team. Not sure why, but when I start the eStreamer client manually, collection seems to continue past the initial lump of events (behaves normally).
For reference, I started it with:

/usr/bin/perl /opt/splunk/etc/apps/eStreamer/bin/estreamer_client.pl -d -c /opt/splunk/etc/apps/eStreamer/local/estreamer.conf -l /opt/splunk/etc/apps/eStreamer/log/estreamer.log
0 Karma

jkleensang
Path Finder

shucks Looks like that solution has a limited lifetime... worked for a couple hours and then died again.

0 Karma

jkleensang
Path Finder

Seems that the root of our problem was that /var filled on the FMC - once that was clear and all services restarted, eStreamer events started flowing again.

0 Karma

dw385
Explorer

How did you clear /var or what is preventing it from happening again?
I don't have direct access to the FMC but I'm told this seems to be our issue too. Trying to work with Cisco for resolution. At the moment they want to delete our estreamer connection and recreate it. I'm not convinced but hope to have this done in the next day or so.

0 Karma

jkleensang
Path Finder

Honestly, I don't know. This device is managed by another team so I don't have any access to it either. It looks like it's just a linux based system, so deleting anything should be pretty straight forward 🙂 There were many mentions/concerns in the release notes for the 6.2.x versions about space, and I think I saw some recommended actions too, so maybe start there if Cisco support doesn't pull through for you.

0 Karma

dw385
Explorer

Also seem to be experiencing the same issue, likely after the 6.2.0.1 upgrade. The debug log from the Splunk estreamer client seems to suggest it connects OK and is "waiting for more logs". I'm told on the FMC there are errors "May 4 18:38:58 FS-95 SF-IMS[18510]: [18510] EventStreamer child(SPLUNK-IP):ConnectionHandler [ERROR] Error sending a message: Not connected"

Although we didn't notice it immediately I do believe it stopped working with 6.2.0.1 upgrade.

Restarting Splunk Estreamer, toggling options doesn't seem to get me more than a few events.
We're configured to log everything but packets.

0 Karma

BHeindel
New Member

I am also experiencing this issue, log files are coming in from Sourcefire and piling up on the Splunk server, just not getting ingested

0 Karma

panovattack
Communicator

when running the search:

search index=_internal "/etc/apps/eStreamer/log" sourcetype!=splunkd_remote_searches

we see the following:

INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/etc/apps/eStreamer/log/estreamer.log.1494267835

do you see the same?

0 Karma

panovattack
Communicator

After a reboot of the FMC, the reference client (latest version) grabs events correctly, however, the estreamer splunk app client still fails after 5 or so events, and only discovery events.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...