topic Re: Indexing FortiGate firewall syslog data, why does Splunk License Usage show 543MB indexed, but our Kiwi syslog server shows 2.7GB for the same data? in Getting Data In

Indexing FortiGate firewall syslog data, why does Splunk License Usage show 543MB indexed, but our Kiwi syslog server shows 2.7GB for the same data?

whitby — Mon, 01 Jun 2015 19:16:46 GMT

I have a FortiGate firewall sending logs via syslog protocol to a Kiwi syslog server on one host, and to Splunk on another host directly via a UDP input.
When I look at the splunk License Usage page it shows that today it indexed 543MB so far, but the Kiwi log file of the same data is 2.7GB so far.
Is Splunk missing some of the data? or is syslogging directly to a Splunk UDP port more efficient in terms of Splunk licensed indexing limits due to something about the syslog protocol that makes it count as less than a straight txt log of the same data??

We are about to buy splunk and thought we needed 4-5GB/day just to account for heavier days of Firewall logs, but if the license usage screen is right it seems that only 1GB would be lots.
If anyone can give me some insight into this large discrepancy between indexing license used when sending syslogs to UDP inputs vs ingesting them as txt files made by kiwi, I would greatly appreciate it.

Re: Indexing FortiGate firewall syslog data, why does Splunk License Usage show 543MB indexed, but our Kiwi syslog server shows 2.7GB for the same data?

bohrasaurabh — Mon, 01 Jun 2015 19:44:34 GMT

We use fortigate too and what I have found that the license usage page in deployment monitor app is accurate. I havent used the License usage in search app. if you run the below search for a week you should get the results very close to the DM app. the result are in Mb.

Re: Indexing FortiGate firewall syslog data, why does Splunk License Usage show 543MB indexed, but our Kiwi syslog server shows 2.7GB for the same data?

bohrasaurabh — Mon, 01 Jun 2015 20:43:54 GMT

I just checked the search app license usage gives the same result as above search, however in bits.

Are you sure the data in kiwi is just for today and not for last 4-5 days?

Re: Indexing FortiGate firewall syslog data, why does Splunk License Usage show 543MB indexed, but our Kiwi syslog server shows 2.7GB for the same data?

jacobwilkins — Mon, 01 Jun 2015 20:55:42 GMT

Sending Syslog directly to Splunk via UDP is your worst option. You are almost certainly dropping events.

You should just put a Universal Forwarder on your syslog server and let it monitor files. That is clean, efficient, and much less likely to fail.

Your indexed license volume should match very closely to the size of the files you are monitoring.

Re: Indexing FortiGate firewall syslog data, why does Splunk License Usage show 543MB indexed, but our Kiwi syslog server shows 2.7GB for the same data?

ppablo — Mon, 01 Jun 2015 21:06:14 GMT

To supplement, here's an awesome previous Answers post on the point brought up by @jacobwilkins that is definitely worth reading, especially the blog by @starcher

http://answers.splunk.com/answers/144357/why-is-syslog-right-into-splunk-so-bad-wrong.html

http://www.georgestarcher.com/splunk-success-with-syslog/

Re: Indexing FortiGate firewall syslog data, why does Splunk License Usage show 543MB indexed, but our Kiwi syslog server shows 2.7GB for the same data?

whitby — Tue, 02 Jun 2015 13:07:25 GMT

Yes, the Kiwi server rolls logs every night at midnight.
It has been recording 3-5GB per day from that firewall for the past several years.
Obviously the Splunk server is not catching everything I'm sending it.

I will have to re-architect the solution to include a syslog close to or directly attached to the firewall with a universal forwarder. I thought that This would work fine as there is plenty of bandwidth between them and the VM I'm running Splunk in has 8 cores and 12GB RAM. Obviously I should keep Syslog on a dedicated syslog machine and indexing/searching on a dedicated Splunk machine.

Re: Indexing FortiGate firewall syslog data, why does Splunk License Usage show 543MB indexed, but our Kiwi syslog server shows 2.7GB for the same data?

whitby — Tue, 02 Jun 2015 13:12:01 GMT

Seems like despite the reasonably high specs I gave the Splunk VM (8 cores, 12GB RAM) Somehow it is missing stuff coming in on the UDP ports (lots of stuff!) so I will try again with a dedicated syslog machine and universal forwarder to the splunk machine that will do just splunk related functions. Thanks guys.
I'll have to talk to our VAR about getting a month extension on our 10GB trial so that I can test this out better and see where we will really stand with all of the things we want to be indexed.