Missing F5 Logs

[IZ88]

Hello fellow Splunkers,

It was brought to my attention from our F5 system manager that some logs from F5 are missing in Splunk. Once every few logs sent (and there are a lot of them) it appears the a log or two just disappear and are not indexed.

To ingest the logs we installed the F5 add-on on our HF and configured both ends (F5 and Splunk) according to the documentation at docs.splunk.com

Does anyone have any idea what can cause this?

We're using Splunk Enterprise 8.0.7 and the Splunk Add-On for F5 4.0.1

Thanks

[PickleRick]

Unfortunately, syslog is typically not a very reliable method of log relaying. Your best bet to find out where the logs are getting lost (is it on the source side or the receiving side) would be to run a tcpdump on the network stream between F5 and your splunk component receiving the syslog data and check whether the "missing" events are there (which would mean that indeed they are probably lost on the splunk's side) or aren't (which would mean that for some reason the events were not sent from the F5 at all.

[IZ88]

Thanks @PickleRick 

I've tried to check where the data disappears.

When looking at the PCAP file from the HF I can see the event, but when searching for it I cannot find it.

Any idea how I can solve it or perhaps an idea for workaround?

[PickleRick]

Are you receiving events directly by tcp/udp input (which one?) on HF?

Do you see any errors on this HF?

[IZ88]

I'm receiving TCP events directly from other sources as well, and the majority of them are not missing events.

The only other source that show the same behavior is Arcsight, but the issue there (according to our support vendor) is that Arcsight sends too much data over one port and syslog-ng can't handle the load, which might explain the occasional missing logs. 

F5 logs are configured as TCP input directly and are not being ingested by syslog-ng.

[PickleRick]

Receiving data directly on splunk's TCP and UDP inputs is not the recommended way to go. But you can try tweaking queueSize and persistentQueueSize parameters. Might help. But then again - might not.

[IZ88]

Hi @PickleRick 

Thank you.

I asked F5 system manager to contact them and see if maybe there's a problem from F5s side.