All Apps and Add-ons

Splunk add-on for opsec (4.0): 2-3 times as much data indexed as version 3

Path Finder

Hi.
I just replaced the old opsec app with the new one (4.0) on our splunk system.
It works fine, but the amount of data indexed is significantly more than the old version from the same source.
I'm looking at 2 or 3 times as much data.
The amount and size of logs on the Checkpoint CLM has not increased. Actually I stopped logging several high-volume rules to try to reduce the amount of logs.

This particular firewall used to log 20-30 GB per day, now it's about 50-80 GB/day.

Doing som digging in the old and new data shows logs from the same CP products (appcontrol, url filter etc.) in the same proportions. So this is probably not a singe product suddenly logging more.

I have not been able to find any duplicates (which was my first thought) but I'm not sure the best way to do a search for duplicates. Any suggestions?

So has anyone else seen something similar? Any suggestions would be appreciated.

0 Karma

Path Finder

We have a very similar issue on OPSEC LEA 4.2.0 and Splunk Core 6.6.0.
Actually, events are logged 13 times and this is obviously destroying our licenses. An easy query to detect the issue is:

sourcetype=opsec*|stats count, dc(_raw) as dedup by host|eval ratio=count/dedup

Strange thing is that a couple of weeks ago, events were indexed 9 times and now 13 times, which means that things get worse over time. Restarting splunk, resetting/recreating OPSEC inputs did not help either.
We have an open case with Splunk...

0 Karma

Path Finder

I'm having the same issue. Were you able to determine if/why you were getting duplicates?

0 Karma

SplunkTrust
SplunkTrust

I would suggest you log a case with Splunk support as this is a Splunk supported application.

In addition you can set the application to "DEBUG" level logging which will provide more output/debugging information in your $SPLUNK_HOME/var/log/splunk/ (the log names include the keyword "checkpoint").
Alternatively, the command it runs can be run from the command line and you can see the raw output that the LEA lo grabber process returns, that might assist you.

Finally, 4.1.0 is out so perhaps you might want to try the latest version ?

0 Karma

Path Finder

Iforgot to mention. This is 4.1.0 on Splunk 6.5

0 Karma

Path Finder

Maybe I shouldn't answer myself, but since this i a rather large post I'll take a chance.

After some digging, its seems like Splunk or the LEA system actually records the logs from Checkpoint three times.

Here is a log record from Checkpoint Tracker, I've removed addresses etc.:
CP logdetail

And the same record when exported with fw log ( I had to slightly modify the format):

31Oct2016 10:00:00 allow removed-ip eth2-01.111 src:removed-ip;dst:193.212.4.120;proto:tcp;appi_name:Google Keep;app_desc:Google Keep is a note-taking service. Google Keep allows to write notes and color code them, to take pictures and record your voice. Supported from: R75.40.;app_id:60460461;app_category:Business Applications;matched_category:Business Applications;app_properties:SSL Protocol, Very Low Risk, Business Applications;app_risk:1;app_rule_id:{0EF98E02-F296-4AAB-AB87-F618F933362F};app_rule_name:default-ut;web_client_type:Other: Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko;app_sig_id:60460461:6;resource:http://clients4.google.com/invalidation/lcs/request;proxy_src_ip:removed-ip;bytes:2357;sent_bytes:19... logs:1;Referrer_self_uid:{58170810-0000-0000-8496-39100C0000C0};user:fjernet;src_user_name:fjernet;snid:51bfa9cd;product:Application Control;service:80;s_port:38975;product_family:Network

While in Splunk the same log looks like this:
Splunk log

The records in Splunk look the same, except for the "loc=" part and the fact that they are cut off at different points. The middle one seems to be the correct one in this sample, while the top and bottom ones are missing data.

Does anyone have a suggestion to where to start looking to figure this out?

Thanks.

0 Karma

Splunk Employee
Splunk Employee

the loc field is a sequential number and assigned to each event logged in the FW database. I'd first compare the events in Check Point to determine if they are actually the same. Would this also happen to be a clustered environment of some sort? ie host=wi_cluster Assuming this could replicate the same event to 3 nodes or separate events?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!