Local udp:514 input not forwarded

[hethu]

Hi,

I have 2 heavy forwarders set up; F1 is forwarding to F2, and F2 forwards to splunk cloud.

On F1 i have set up a local input to listening on UDP:514 for events, this works great and forwards to cloud.
On F2 i have set up a local input for UDP:514 exactly like i did on F1, but no events are forwarded, does anyone here have a clue to what could be wrong?

The events are of the same type, so as long as this works on F1 it should not be an issue with interpreting/reading the events.

I have checked the FW and the events are beeing received, and also after setting UDP processor log level to debug i get this in my splunkd.log on F2:

02-01-2021 12:54:00.520 +0100 DEBUG UDPInputProcessor - callback()
02-01-2021 12:54:10.512 +0100 DEBUG UDPInputProcessor - callback()
02-01-2021 12:54:18.502 +0100 INFO  TcpOutputProc - Found currently active indexer. Connected to idx=ForwarderIP:30132, reuse=1.
02-01-2021 12:54:20.467 +0100 DEBUG UDPInputProcessor - Generating UDP metrics
02-01-2021 12:54:20.467 +0100 DEBUG UDPInputProcessor - callback()
02-01-2021 12:54:30.514 +0100 DEBUG UDPInputProcessor - callback()
02-01-2021 12:54:34.790 +0100 DEBUG UDPInputProcessor - event=data from="PC100.Local (new)" status=accepted
02-01-2021 12:54:34.790 +0100 DEBUG UDPInputProcessor - UDPInputProcessor::when_events called
02-01-2021 12:54:34.801 +0100 DEBUG UDPInputProcessor - event=data from=PC100.Local status=accepted
02-01-2021 12:54:34.801 +0100 DEBUG UDPInputProcessor - UDPInputProcessor::when_events called
02-01-2021 12:54:34.812 +0100 DEBUG UDPInputProcessor - event=data from=PC100.Local status=accepted
02-01-2021 12:54:34.812 +0100 DEBUG UDPInputProcessor - UDPInputProcessor::when_events called
02-01-2021 12:54:34.830 +0100 DEBUG UDPInputProcessor - event=data from=PC100.Local status=accepted
02-01-2021 12:54:34.831 +0100 DEBUG UDPInputProcessor - UDPInputProcessor::when_events called
02-01-2021 12:54:44.829 +0100 DEBUG UDPInputProcessor - callback()
02-01-2021 12:54:44.829 +0100 DEBUG UDPInputProcessor - event=sendDoneKey source=PC100.Local localport=514
02-01-2021 12:54:44.829 +0100 DEBUG UDPInputProcessor - event=deleteSource source=PC100.Local localport=514
02-01-2021 12:54:48.413 +0100 INFO  TcpOutputProc - Found currently active indexer. Connected to idx=ForwarderIP:30132, reuse=1.
02-01-2021 12:54:50.471 +0100 DEBUG UDPInputProcessor - Generating UDP metrics
02-01-2021 12:54:50.471 +0100 DEBUG UDPInputProcessor - callback()

 I have had to replace some hostnames as you probably can see. Hopefully someone here can help me figure this out.

[hethu · Accepted solution]

It seems the input i set up through the web interface, did not change the active inputs.conf.... after i manually altered this config file, the forwarder correctly received and forwarded my events.

[richgalloway]

To help future readers, please describe the manual changes you had to make.

[richgalloway]

Heavy forwarder F2 should be listening on port 9997 for the data from F1.

The use of intermediate forwarders like F2 is discouraged.  Forwarders should send data directly to indexers.  Having another forwarder in the path can lead to unbalanced data on the indexers, can be a bottleneck, and is an extra layer to manage and troubleshoot.