Support for high volume eStreamer data transfer

[dkmcclory]

Our Splunk ingestion for eStreamer events appears to be getting overwhelmed by the amount of data we receive.  Currently, our ingestion averages over 10,000 events per second, and Cisco support indicate that the existing Splunk app that we've been using cannot handle that volume.  Is there an approach we can use to support that interface with a volume at that level?

Currently, we are using this app:
https://splunkbase.splunk.com/app/3662

Should we be using this app?
https://splunkbase.splunk.com/app/7404

And, if we switch apps, will the new one (7404) be able to keep up with the data transfer volume?

[dkmcclory]

An update for those who may have similar issues:

Support for these Technical Apps and the eStreamer protocol interface to Splunk is not provided by either Splunk Support or Cisco Support.  Opening tickets with either party did not successfully resolve these questions.  The only support currently available for this interface is to use the email splunk_cisco_security_cloud@cisco.com.  If staff are available to respond, they will answer some limited questions.

Cisco Security confirms that Splunkbase #3662 (TA-eStreamer) has been desupported in 2024, because the eNcore client is it based upon has been desupported.  In essence, the eNcore client is being replaced.  As a result, it is recommened (by Cisco Security) that systems with Splunkbase #3662 replace it with the new (supported) TA, which is Splunkbase #7404 (Cisco Security Cloud).  

Cisco Security indicates that the older TA-eStreamer had a maximum expected throughput of under 10,000 events per second, and practically limited to under 8K of continuous throughput.  So, applications like our that routinely exceed 8K per second would never have successfully used TA-eStreamer for that performance level.  Performance levels for the newer Cisco Security Cloud app (#7404) are expected to be in the maximum range of 15-20K events per second, because it uses a new eStreamer SDK that replaces to decommissioned eNcore client software.  Since it is possible our application may rise above that level, I asked if the app potentially supported using things like load balancers to scale beyond 15-20K, but no answer was provided for this question.

The Cisco Security team responding to my questions indicated that support for this TA is somewhat limited, with only a best effort support during Eastern US office hours.

The Cisco Security team of course also recommends using hardware that follows their documented guidelines and provides sufficient memory, disk, and CPU to run the software at its maximum performance levels.  But, once those maximum performance levels are reached (15-20K per second) there is no recommended scaling beyond that.  

Our team expects to experiment in various setups and architectures to see how far we can push the new TA.

[PickleRick]

I'm not saying that it's great news since the support is limited and so on but it's good to know that at least _someone_ has at least _some_ knowledge of what's going on 😉

Thanks for sharing.

[PickleRick]

Application obsolescence is one thing. Performance is another (yes, there can be differences in performance between those apps but still as one is getting obsoleted it might soon be difficult to find support for it).

Unless you have just one big source (in which case it might indeed be difficult to scale up your environment to meet the demands for performance) you might want to consider splitting your eStreamer sources between different HFs so that you don't have a single component overwhelmed with inputs.

[dkmcclory]

There is indeed just one big source, because the Cisco FMC concentrates all Cisco logs and then provides the transfer interface via eStreamer.

[kiran_panchavat]

@dkmcclory The Cisco Secure eStreamer Client Add-On for Splunk (App ID: 3662) has been marked for end-of-life as of July 15, 2024, with limited support available. Users are advised to transition to the Cisco Security Cloud App for Splunk (App ID: 7404), which integrates the eStreamer SDK to provide comprehensive event support, including IDS, Malware, Connection, and IDS Packet data.

[dkmcclory]

Indeed, Cisco Security (the supporting team) provided additional information that explained in detail what was happening with the TAs involved.  Please see my comments above.

[kiran_panchavat]

@dkmcclory 

To improve ingestion performance independently of the app or add-on used, consider optimizing your hardware resources, such as CPU and RAM. Allocating additional CPU cores and memory can significantly enhance the handling of high event rates. Ensure a stable, high-speed network connection between the Firepower Management Center (FMC) and the Splunk to avoid data bottlenecks. Use high-performance storage solutions to manage the rapid write operations required for high event volumes. Additionally, filter out unnecessary events in the Heavy forwarder by configuring props.conf and transforms.conf before sending data to indexers.

[dkmcclory]

Cisco provides a recommendation for hardware that should be followed but there are no recommended hardware or architecture setups that will allow the older TA to scale beyond an 8k throughput.  Only an upgrade to the new TA will allow a higher throughput.