I'm trying to prevent unnecessary sprawl in our current splunk environment and I want to funnel all udp:514 traffic from systems that cannot change port at point of departure to a heavy forwarder that will load balance across indexers. I'm running into some challenges in doing so based on how different splunk apps treat the port.
Cisco IOS: looks for sourcetype=syslog and transforms and writes the data to the ios index.
Splunk app for Netapp Data OnTAP: needs udp:514's sourcetype to be ontap:syslog
How could I effectively have apps written to use syslog ports differently work together on the same heavy forwarder if they require the default sourcetype to be different at the point of arrival?
Perhaps you can simply filter the events by source IP or hosts in inputs.conf and route that to different sourcetypes. Look the inputs.conf example bellow:
[udp://10.1.0.252:514] connection_host = ip source = asa_firewall_headquarters sourcetype = syslog index = cisco [udp://10.1.3.251:514] connection_host = ip source = fortigate_firewall_branch sourcetype = fortigate index = fortigate
Hope this can help you!
gfreitas' answer is a way to go, or you could set up a Syslog-NG/Rsyslog server for reception of syslog data to file. You then set up a Universal Forwarder to monitor these files and sending them to the indexer.
Both Syslog-NG and Rsyslog can do per host filtering. Another way to go is to use different ports or even different IP addresses by setting up secondary IPs on that server and treating the inputs differently by IP.
My suggestion is normally always to set up a pure syslog server. That way no data is lost when Splunk is restarted due to patching, configuration changes an so on.