topic Re: EPS in flat file with universal forwarder in Splunk Search

EPS in flat file with universal forwarder

Meet-Patel — Wed, 10 Apr 2024 17:30:15 GMT

Hi Team,

what is the Events-per-second (EPS) in flat file with universal forwarder?

Re: EPS in flat file with universal forwarder

richgalloway — Wed, 10 Apr 2024 18:16:57 GMT

It depends on the size of an event. The UF is rate-limited by the maxKBps setting in limits.conf.

Re: EPS in flat file with universal forwarder

PickleRick — Wed, 10 Apr 2024 18:58:15 GMT

It's a bit more complicated than that.

1. As @richgalloway pointed out, UF is by default capped with maxKBps (which is a rough value - there is no guarantee for Splunk to _always_ process no more than that value per second).

2. Even if you set the limit to 0 (no limit at all), the back pressure from output will make the forwarder to stop reading the file until the queue empties a bit.

Generally, the "speed" of Splunk reading files depends mostly on non-Splunk limits (like output rate which might be limited by receiving instance performance or network bandwidth or input rate if the file is placed on a network share). Also since the limits apply to the general overall size of the data regardless of how big the events are, the EPS value isn't that important here - the same limit will apply if you send just a few big events as when you send many small ones.

But there is also one more thing worth pointing out - UF doesn't (typically, unless you use indexed extractions on structured data) deal with events as such - it reads and sends to an output chunks of data for breaking into events "further down the road" (on indexers or heavy forwarders). With sufficiently modern UF and configured EVENT_BREAKER, you should be sending chunks of data ending on event boundary, but you typically don't send single events (unless they are huge).

Re: EPS in flat file with universal forwarder

Meet-Patel — Thu, 11 Apr 2024 06:36:13 GMT

I would appreciate it if there were any documents on Events-per-second (EPS) recorded in a flat file with universal forwarder.

Re: EPS in flat file with universal forwarder

PickleRick — Thu, 11 Apr 2024 08:16:14 GMT

What is your business problem?

Re: EPS in flat file with universal forwarder

Meet-Patel — Thu, 11 Apr 2024 11:03:42 GMT

We want to read log files (approx. 100 of GBs) and send them through Splunk forwarder before setting up, We need to verify the Events-per-second (EPS) recorded in a flat file with Universal Forwarder.

Re: EPS in flat file with universal forwarder

PickleRick — Thu, 11 Apr 2024 12:28:42 GMT

OK. Have you read anything that has been written in this thread? EPS as such is not a very important concept for Splunk (at least not on the UF level).

Re: EPS in flat file with universal forwarder

richgalloway — Thu, 11 Apr 2024 12:29:50 GMT

I find it interesting that you give the log file size in GB rather than events yet you expect UF documentation to provide EPS.

@PickleRickhas explained why we cannot offer an EPS number and also why any talk about data rates is a guess at best.

A Splunk UF is very capable of handling 100GBs of log files. Many customers do so regularly.

Why problem are you trying to solve?

Re: EPS in flat file with universal forwarder

PickleRick — Thu, 11 Apr 2024 13:18:59 GMT

Anyway, obsessing about EPS suggests that you might be thinking about replacing some other SIEM/log management solution. Those used to be licensed on a per-EPS basis. With Splunk it doesn't matter. If ingest-based your license allows for indexing specified volume of data _daily_ regardless of whether it's a constant steady data stream or if it's just a few "batches" of high volume peaks of data.

Re: EPS in flat file with universal forwarder

Meet-Patel — Fri, 12 Apr 2024 12:14:45 GMT

I am looking for something similar to this.

https://docs.splunk.com/Documentation/WindowsAddOn/8.1.2/User/PerformancereferencefortheSplunkAdd-onforWindows

Re: EPS in flat file with universal forwarder

richgalloway — Fri, 12 Apr 2024 14:05:54 GMT

That document is for a specific source where the event size is well-defined. The information there cannot be generalized because the size of an "event" is unknown. I've seen event sizes range from <100 to >100,000 bytes so it is very difficult to produce an EPS number without knowing more about the data you wish to ingest.

It's possible the documentation for other TAs provides the information you seek. Have you looked at the TAs for your data?

Re: EPS in flat file with universal forwarder

PickleRick — Fri, 12 Apr 2024 14:21:12 GMT

Apart from all that @richgalloway already mentioned, this document shows results of testing on some particular reference hardware. It's by no means a guarantee that an input will work with this performance.

Also remember that windows eventlog inputs get the logs by calling the system using winapi whereas file input just reads the file straight from the disk (most probably using memory-mapped files since it's most effective method).

And last but definitely not least - as I already pointed out - UF typically doesn't break data into events!