How to optimize the ingestion of data from Splunk Universal Forwarder to Splunk Indexer

sdhiren — Thu, 14 Mar 2024 11:16:07 GMT

I have a splunk universal forwarder, which is indexing a 1 GB log file to a Splunk Indexer. The problem I am facing is the ingestion is happening very slow (100K log entries per mins). I have tried setting the

parallelIngestionPipelines = 2

setting for both Indexer and Forwarder, but to no avail.

Below are the stats for the containers running Indexer and forwarder

CONTAINER_ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
ecb272b9ca6b tracing-splunk-1 12.15% 260.8MiB / 7.674GiB 3.32% 366MB / 1.85MB 0B / 1.01GB 239

CONTAINER_ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
0ac17f935889 tracing-splunkforwarder-1 0.70% 68.22MiB / 7.674GiB 0.87% 986kB / 312MB 0B / 18.2MB 65

We are running these in a docker container

I and my team is pretty new to Splunk eco system. Can someone please help us to optimize the ingestion of logs.

Re: How to optimize the ingestion of data from Splunk Universal Forwarder to Splunk Indexer

richgalloway — Thu, 14 Mar 2024 12:23:18 GMT

Make sure you have this in limits.conf on the UF

[thruput] maxKBps = 0

Re: How to optimize the ingestion of data from Splunk Universal Forwarder to Splunk Indexer

sdhiren — Thu, 14 Mar 2024 13:04:24 GMT

thanks @richgalloway with that change, around 5 million logs were ingested in couple of mins.

topic Re: How to optimize the ingestion of data from Splunk Universal Forwarder to Splunk Indexer in Getting Data In

How to optimize the ingestion of data from Splunk Universal Forwarder to Splunk Indexer

Re: How to optimize the ingestion of data from Splunk Universal Forwarder to Splunk Indexer

Re: How to optimize the ingestion of data from Splunk Universal Forwarder to Splunk Indexer