All Apps and Add-ons

Cisco eStreamer for Splunk: connection event/flow logs delay

BHeindel
New Member

Searching eStreamer data in Splunk - it appears that most data seems to come in fairly quickly, almost real time (file/malware) - but for connection event/flow data it appears to lag behind - sometimes an hour later.

Is this something that others have noticed - or unique to our environment.

Firepower 6.2.0.1
eStreamer 2.2.2

0 Karma

gordo32
Communicator

I know this is an old thread, but since a google search led me to it, others will probably read it too. This solution is defined in this other article: https://community.splunk.com/t5/All-Apps-and-Add-ons/Cisco-eStreamer-eNcore-delay-in-logs-getting-to...

TL;DR (or in case the URL above breaks), by default eStreamer only picks up events if there are 100 or more to collect. On low-volume systems, this could be an issue, so there is a batchSize parameter to adjust. The end of my estreamer.conf now has this added line at the end:

"workerProcesses": 4,
"batchSize": 5
}

0 Karma

douglashurd
Builder

If you are using Firepower 6.x then you should use this TA: https://splunkbase.splunk.com/app/3662/ v 3.5.4

And you should use this version of the Dashboard: https://splunkbase.splunk.com/app/3663/ V 3.5.3

2.2.2 is a combined App and TA for Firepower 5.4 customers. It's not going to work well for 6.x customers.

Doug

0 Karma

Anonymous
Not applicable

I our enviroment, I have experienced the FMC server to send different logs in bulks.
Since the event has a timestamp i think it is by design.

0 Karma

douglashurd
Builder

Highly recommend you use the new eNcore for Splunk add-on for Firepower 6.x.

This is s a complete re-write estreamer client built in Python with a Splunk plugin.

https://splunkbase.splunk.com/app/3662/

Its able to scale with more CPU/RAM and supports the entire 6.x schema, delivering fully qualified event data.

Doug

0 Karma

ejmin
Path Finder

Hi Sir Douglas were currently experiencing an error issue regarding your estreamer app with version 2.2.2 and the latest version which you stated above. We'd also followed the splunk version requirements in your app our client's FMC version is 6.1.0.4 and when we configuring both of your app 2.2.2 and the following versions we always directing to this error IO::Socket::INET6 configuration failederror:140E0197:SSL routines:SSL_shutdown:shutdown while in init" I dont know why but our OpenSSL version is up to date and all the required perl modules are installed. We search for this error for a week but we still didnt connect it to the FMC od our client. Would greatly appreciate if you will reply to this post.

0 Karma

panovattack
Communicator

We've tested 6.2.2 and it works as well.... how do we leverage "Multi-Process Design: Will scale with additional compute resources to support event rates"

We can only seem to get encore to use one processor and process. We added CPU specifically to support the Multi Process design. A setting to change does not jump out at us.

0 Karma

douglashurd
Builder

You will also want to patch 6.2.0.1 to 6.2.0.5 to address an estreamer bug.

0 Karma
Get Updates on the Splunk Community!

New Case Study Shows the Value of Partnering with Splunk Academic Alliance

The University of Nevada, Las Vegas (UNLV) is another premier research institution helping to shape the next ...

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...