Kiwi syslog server and Palo Alto app for splunk issues

[Iwdavies]

We have 3 palo alto firewalls that I'm sending syslog data to a solarwinds kiwi syslog server.  I am having kiwi write the logs to disk and have the splunk universal forwarder send the logs to my splunk environment.  I have 2 issues which I can't seem to figure out (even after looking at various posts here that mention similar scenarios).

1)  The Palo Alto app states that there is only 1 firewall.  When i look in the logs that "firewall" is the kiwi syslog server.

     I have tried adding a Host variable to the inputs (no such luck).  Tried having kiwi just forward the logs               directly to splunk (no such luck).  I have even tried to have kiwi just send the raw data.  In every case kiwi         is appending its own date / time stamp and host value in front of the palo alto messages.  I'm not sure               how to completely strip that information.

2)  The Palo Alto see's the source type as Pan:Logs, but it is not seperating them into their perspective logs:  i.e. Pan:Firewall, Pan:System, Pan:traffic etc..

     I have seen suggestions to use a transforms file and / or a props file, but I'm just too new to understand             how to configure them properly.

Any help would be appreciated.

[Richfez]

The Kiwi Syslog Server is something many people have problems out of.  Indeed, the Venn diagram of "people who use kiwi" and "people who have problems with kiwi" is nearly two overlapping circles.

Possible solution if you stick with Kiwi

It might be that there are some kiwi syslog settings you can change.  It appears there's a whole section in the admin manual about log file and database formats.

What to change it to?  Well, I'd skip anything with the word "Kiwi" in it, because those are of course all non-standard "kiwi specific" file formats, which means nothing else understands them.

Maybe the BSD style log format will work?  You could give it a try, won't be any worse than what you have now.

Though at some point you'll hit Kiwi's scaling limits, which I hear are very, very low.

Better Solutions

I don't really mean to disrespect Kiwi Syslog Server, but it's really been a problem for a lot of people.

If it were me, I'd stand up a small virtual machine running Linux or pretty much any distribution, remove rsyslog if it has it (Just way harder to configure due to a really obtuse syntax), install syslog-ng and google for the configs you'll want there, put on the Universal Forwarder configured to send its output to your indexer and ... bask in the gloriously well-working, nearly bullet proof zero maintenance syslog server that works in a standard, predictable and compatible way.

Heck, I bet you could run syslog-ng in Ubuntu on WSL, right there "inside windows".  Try that - get Window's Ubuntu up on WSL, and do a "sudo apt install syslog-ng" and go from there.  You MIGHT have to configure it to listen to a higher port instead of 514 (because... well, reasons, possibly), but even with that, it's worth a try.

Happy Splunking!

-Rich

[Iwdavies]

After looking at this from a kiwi side, I realized that it doesn't matter how much data kiwi adds to the stream.  Even if I were to be able to remove all the kiwi header, splunk would still think the host is only 1 server (the kiwi server) since only one server is technically sending data to kiwi.  So my real issue is to get splunk to recognize the host from the log file and not from the sending device.  I understand that to do this I need to use a transforms and props file, but am still very confused on how to accomplish this.

[Iwdavies]

So apparently I can strip data out of the syslog file before it is sent to splunk.  This will help deal with 1 of my two issues.  Here is an example of the start of each row in the log file:

                                   host                                                        serial #

Oct 07 15:01:42 x.x.x.x 1,2020/10/07 15:01:42,xxxxxxxxxxxxxxx,TRAFFIC

To more clearly state my goals:

1)  I would love to strip the 1st date/time stamp (as you can see there are 2) from the data before it is indexed.

2)  I would love to have splunk identify the host as either the ip address of the host of the serial # of the host instead of the syslog server that is sending this data to splunk

3)  Currently the universal forwarder that is sending this data to splunk is using a sourcetype of Pan:logs, but it is not splitting out the logs into their appropriate subcatagory (i.e. Pan:firewall, Pan:System, Pan:traffic etc).

Ian