topic Re: UF versus HF processing in Getting Data In

UF versus HF processing

craigkleen — Fri, 29 Jan 2021 22:52:01 GMT

Currently, my firewall logs (PaloAlto) are sent via syslog to a virtual Linux machine. On that machine, I run a full version of Splunk (Heavy Forwarder 8.x) that sends into separate indexers.

I was planning to migrate the syslog data to new Linux servers and use Universal Forwarder instead, but running into what looks like some serious performance issues. The UF will send a big chunk of data to start, but then the index stops receiving from the UF.

I tried the post at https://community.splunk.com/t5/Getting-Data-In/Universal-Forwarder-ParsingQueue-KB-Size/td-p/50410 to increase the size of the parsingqueue, but that didn't help.

I'm not quite sure what to look at next. Maybe the stream is too much for UF to handle? I haven't found anything definitive on that subject.

Re: UF versus HF processing

isoutamo — Fri, 29 Jan 2021 22:58:31 GMT

on UF are you receiving syslog via native syslog and then reading those from file or directly UF’s udp/tcp listener?

r. Ismo

Re: UF versus HF processing

craigkleen — Fri, 29 Jan 2021 23:00:15 GMT

On both, that's the usual process. Native "rsyslog" daemon writing to a file, and UF then reading that.

Re: UF versus HF processing

isoutamo — Fri, 29 Jan 2021 23:02:54 GMT

And how much you have that traffic (EPS + size)? Which kind of host/fs/IOPS? And is the HF equal with UF?

Re: UF versus HF processing

craigkleen — Fri, 29 Jan 2021 23:09:28 GMT

From a machine standpoint, the HF and UF are the same. Both are virtual servers that are clones of each other. The only difference is the Linux version (going from RHEL6 to RHEL8).

If I splunk: host=HF index=_internal eps=* group=per_source_thruput source=panfwlog

The max EPS I get is right around 1,200.

A similar search with host=UF during the time the firewall is sending to this new server, is showing me EPS under 4? Super weird.

The data is getting written to disk, and when I switch the firewall back to the old server, the UF eventually does catch up, but it's not reading like the HF does.

Re: UF versus HF processing

scelikok — Sat, 30 Jan 2021 06:29:59 GMT

Hi @craigkleen,

Are you using the same outputs.conf and limits.conf on both servers? UF has default bandwidth limit for 256KB/s. Since HF does not have this limit, you have to add this on UF instance.

limits.conf [thruput] maxKBps = 0

Re: UF versus HF processing

craigkleen — Mon, 01 Feb 2021 18:25:42 GMT

That was the ticket.

Under ${SPLUNK_HOME}/etc/system, the limits.conf was the same. But, on the UF, under ${SPLUNK_HOME}/etc/apps/SplunkUniversalForwarder/ the default was overridden to 256K. So, I made a local directory and updated there.

So, thanks for the pointer!