I have a Kiwi Syslog server where all Cisco logs are getting stored in folders with the host IP as the folder name.
I have configured an app in the universal forwarder that reads those logs. Below is the inputs.conf,
[monitor://E:\SyslogData\10.10.10*\log*] host_segment = 2 index = test_index sourcetype = syslog disabled = 0
When the Kiwi log file format is set to Kiwi format ISO yyyy-mm-dd (Tab delimited) while data encoding is UTF-8. The host value extracted is Local0.Info.
And with the above settings, the logs in kiwi look like below :
2018-08-01 16:06:30 Local0.Info 10.10.10.X.X 1 153232424240.525452732241 cfC_G_AP07 flows allow src=220.127.116.11 dst=18.104.22.168 mac=43:43:5C:83:25:CA protocol=t3p sport=7240 dport=6224
The only exception when the extraction with the above inputs.conf works well is if the log format is changed in Kiwi to either "message text only (no priority) or raw logging", however, even in that case the data does not get populated in the Cisco Networks app. It still shows no data.
Last thing I tried was I uninstalled my custom app I had deployed in UF that had the above inputs.conf and deployed Cisco Networks add-on in the UF with the above inputs.conf, in the hope that maybe transforms.conf from the add-on would properly parse data and send to Indexer. But after doin that, I have stopped receiving data altogether in Splunk.
Arch Info : 1 S.H, 1 Indexer, 1 HF
I have tried numerous way to try to make this work but nothing has helped. Please help
Try and configure Kiwi to write the full raw log as it received it from the network. That should ensure it is in the format as the TA expects it.
When using sourcetype=syslog, you automatically get the syslog hostname extraction transform for free, that Splunk has built in in some default config file. Either use the cisco TA's specific sourcetype, or override the syslog hostname transform to block that host field extraction (since you are already populating the host field using the host_segment setting, so you don't want that to get overwritten). You can do that by adding the following to a local props.conf file.
[source::E:\SyslogData\10.10.10*\log*] TRANSFORMS =
Which prevents the
TRANSFORMS = syslog-host from system/default/props.conf from getting applied.
Configured Kiwi log file format as raw logs
Yes, the add-on has been deployed literally everywhere, the S.H, indexer, heavy forwarder.
The data is in this format now,
<134>1 14245934.01624555 tfC_G_Af14 flows allow src=22.214.171.124 dst=126.96.36.199 mac=50:5E:55:53:82:30 protocol=tcp sport=3560 dport=56
Thats the Cisco PIX PFSS format (raw logging) set in Kiwi
When I look at the props and transforms files of the add-on, that doesn't include anything that seems to match these type of logs.
I'm wondering if this particular cisco product is supported by the add-on, or whether there is still something weird going on with the log format somehow.