Is the collection script for the Splunk Add-on for Check Point OPSEC LEA just pulling the log files or is it pulling events over an API directly?

[btiggemann]

Hi Splunkers,

We have a customer that is collecting Check Point fw, ips, and vpn logs via Opsec. Check Point version is R77.
At the moment, Splunk is indexing about 30 gigabyte per day. If we look at the log directory at Check Point smartcenter, we only see something about 3 gigabytes (rotating every 2GB) for the specific day, but Splunk has indexed 30 gigabytes.

I found out that Check Point logs are written in binary, but are they also saved in a compressed way?

Does anybody know how the OPSEC script from Splunk is pulling the logs? Is it just reading the files or is there an APIcall directly to the smart center? How can we check why we have this gab between the logs files on the system and the indexed log volume?

Thanks in advance

[btiggemann]

I have done some research at this topic...

What the OPSEC TA will do:
It will run a script that may is referencing to the internal log files stored at the disk or it will pull everything with the special 'fw log' command.
But that doesn't matter at all. Because the information that is stored in the log files are csv based and compared to the OPSEC TA data, most of the fields in the log files are missing.

Here is an example:

1. The log file on the Check Point management server is CSV based, there is no overhead from writing key value pairs in the event. Writing key=value in each event will increase the log volume.
2. Lots of information are missing in the log file, as you can see, the data from the OPSEC TA script is enricht with some additional fields. Thats the second point that causes more log volume.

At the end these two things are the reason why there is such a big difference between the size of log files on the disk of the CP Smart Center and those generated via TA OPSEC.

It will definitely be helpful if TA OPSEC will provide a delimiter based event structure just to save some volume.
But at the other hand I have heard that Check Point will offer Syslog funtionality in the future and the whole OPSEC thing will be obsolete.

If you are good at coding, you can change the TA OPSEC script to generate a shorter delimiter based format as well.

[alemarzu]

@ btiggemann, Thank you so much for your time and explanation.

[alemarzu]

Sup mate, I'm facing the same issue here.

Compressed files on Checkpoint SC are roughly 5~6 GBs per day but Splunk is consuming 20 GBs +/- per day. Any explanation for this behavior?

KR.

[btiggemann]

Hi KR,
I can write down what I have found out in my research.
You can have a look in a few minutes.
best regards
Benjamin

[Chubbybunny]

from what I understand, the logs are stored in a proprietary format that can only be accessed by using OPSEC or the 'fw log' command.

If you happen to be finding inconsistencies in gathering FW log data from the Check Point manager, you may want to open a support case to have an AWESOME Engineer check for underlying issues with the App o Forwarder.

[alemarzu]

@Chubbybunny

Actually I've performed cross-data tests between Splunk and Checkpoint Managent Console and it seems to be fine, no inconsistencies atm.

Thanks.