Splunk Search

EPS in flat file with universal forwarder

Meet-Patel
Loves-to-Learn Lots

Hi Team,

what is the Events-per-second (EPS) in flat file with universal forwarder?

Labels (1)
Tags (2)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

It's a bit more complicated than that.

1. As @richgalloway pointed out, UF is by default capped with maxKBps (which is a rough value - there is no guarantee for Splunk to _always_ process no more than that value per second).

2. Even if you set the limit to 0 (no limit at all), the back pressure from output will make the forwarder to stop reading the file until the queue empties a bit.

Generally, the "speed" of Splunk reading files depends mostly on non-Splunk limits (like output rate which might be limited by receiving instance performance or network bandwidth or input rate if the file is placed on a network share). Also since the limits apply to the general overall size of the data regardless of how big the events are, the EPS value isn't that important here - the same limit will apply if you send just a few big events as when you send many small ones.

But there is also one more thing worth pointing out - UF doesn't (typically, unless you use indexed extractions on structured data) deal with events as such - it reads and sends to an output chunks of data for breaking into events "further down the road" (on indexers or heavy forwarders). With sufficiently modern UF and configured EVENT_BREAKER, you should be sending chunks of data ending on event boundary, but you typically don't send single events (unless they are huge).

 

0 Karma

Meet-Patel
Loves-to-Learn Lots

I would appreciate it if there were any documents on Events-per-second (EPS) recorded in a flat file with universal forwarder.

Tags (2)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

What is your business problem?

0 Karma

Meet-Patel
Loves-to-Learn Lots

We want to read log files (approx. 100 of GBs) and send them through Splunk forwarder before setting up, We need to verify the Events-per-second (EPS) recorded in a flat file with Universal Forwarder.

Tags (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

I find it interesting that you give the log file size in GB rather than events yet you expect UF documentation to provide EPS.

@PickleRickhas explained why we cannot offer an EPS number and also why any talk about data rates is a guess at best.

A Splunk UF is very capable of handling 100GBs of log files.  Many customers do so regularly.

Why problem are you trying to solve?

---
If this reply helps you, Karma would be appreciated.
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Anyway, obsessing about EPS suggests that you might be thinking about replacing some other SIEM/log management solution. Those used to be licensed on a per-EPS basis. With Splunk it doesn't matter. If ingest-based your license allows for indexing specified volume of data _daily_ regardless of whether it's a constant steady data stream or if it's just a few "batches" of high volume peaks of data.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK. Have you read anything that has been written in this thread? EPS as such is not a very important concept for Splunk (at least not on the UF level).

0 Karma

Meet-Patel
Loves-to-Learn Lots
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Apart from all that @richgalloway already mentioned, this document shows results of testing on some particular reference hardware. It's by no means a guarantee that an input will work with this performance.

Also remember that windows eventlog inputs get the logs by calling the system using winapi whereas file input just reads the file straight from the disk (most probably using memory-mapped files since it's most effective method).

And last but definitely not least - as I already pointed out - UF typically doesn't break data into events!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

That document is for a specific source where the event size is well-defined.  The information there cannot be generalized because the size of an "event" is unknown.  I've seen event sizes range from <100 to >100,000 bytes so it is very difficult to produce an EPS number without knowing more about the data you wish to ingest.

It's possible the documentation for other TAs provides the information you seek.  Have you looked at the TAs for your data?

---
If this reply helps you, Karma would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

It depends on the size of an event.  The UF is rate-limited by the maxKBps setting in limits.conf.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...