We have a Universal Forwarder on a Domain Controller (DC) that is forwarding all the local logs to a 4.1.7 Forwarder. The 4.1.7 Forwarder is then sending the logs to an Indexer, as well as an IDS via syslog. This 4.1.7 Forwarder is also collecting tons of WMI logs which are being observed on both the Indexer and IDS.
We are seeing the DC logs come across to the indexer however we are not seeing any of the DC logs go to the IDS. As the Universal Forwarder sends cooked data I tried setting cooked data to false:
This is the Universal Forwarder config. /etc/system/local/outputs.conf
[tcpout] defaultGroup = splunk02..._9998 disabled = false indexAndForward = 0 [tcpout:splunk02..._9998] server = splunk02...:9998 [tcpout-server://splunk02...:9998] sendCookedData=false
After this change I was still able to observe DC logs on the Indexer however none on the IDS. For troubleshooting purposes i installed a LightForwarder on the DC and was able to see DC logs on both the Indexer and the IDS. This leads me to believe that the data is getting cooked by the Universal Forwarder. Anyone have any ideas on how to make the Universal Forwarder send data unCooked or see what im doing wrong here?
Here is the config on the 4.1.7 Forwarder /etc/system/local
[tcpout] disabled = false indexAndForward = false [syslog:my_syslog_group] disabled = false server = 10.x.x.x:514 type = udp sendCookedData = false
[host::*] DATETIME_CONFIG = NONE TRANSFORMS-ROUTING = send_to
[send_to_AG] REGEX = . DEST_KEY = _SYSLOG_ROUTING FORMAT = my_syslog_group
UDP forwarding is not supported on the Universal Forwarder. It's allowed only on the Heavy Forwarder.
From the documentation: You can configure a heavy forwarder to send data in standard syslog format. The forwarder sends the data through a separate output processor. You can also filter the data with props.conf and transforms.conf. You'll need to specify _SYSLOG_ROUTING as the DEST_KEY.
Note: The syslog output processor is not available for universal or light forwarders.
The syslog output processor sends RFC 3164 compliant events to a TCP/UDP-based server and port, making the payload of any non-compliant data RFC 3164 compliant. Yes, that means Windows event logs!
To forward syslog data, identify the third-party receiving server and specify it in a syslog target group in the forwarder's outputs.conf file.
Note: If you have defined multiple event types for syslog data, the event type names must all include the string "syslog".
Forward syslog data
In outputs.conf, specify the syslog target group:
Have you tried the following stanza in outputs.conf?
[tcpout-server://splunk02...:9998] server = 10.255.4.213:514 sendCookedData=false
I am just basing that off the documentation for forwarding data found here. I am not entirely certain how you want to send data but if its a subset of syslog data then you may be interested in the section near the end (found here)
No luck but thanks for the suggestion. I spoke with Splunk support regarding this and their best guess was that there is an issue with the 4.2 UF sending to the 4.1.7 forwarder. They suggested that I upgrade the forwarders when 4.2.1 is available in a week or two and see if that works.