All Apps and Add-ons

Splunk Add-on for Check Point OPSEC LEA: Why is the same data indexed multiple times?

splunker12er
Motivator

Checkpoint firewall log storage policy is set to rotate everyday at midnight where new "file id" is created for the log file "fw.log" every midnight.

In the Splunk forwarder -
For every new file id - splunk app creates a new stanza under the file : opsec-log-status.conf

Eg:

[1438127971@LEA10.250.200.232]
fileid = 1438127971
filename = 2015-07-29_235900.log
last_rec_pos = 70000

[1438214351@LEA10.250.200.232]
fileid = 1438214351
filename = fw.log
last_rec_pos = 35441

Every time the script triggers, the old file id (which has been already collected data t splunk) also runs and gets the same data again to Splunk. This causes high license usage.

Only the latest file id should be should be monitored with the last loc values it has left.

Check point device maintains the log track file clearly, where every day an entry is made (fw.logtrack) with the current status.

Why doesn't Splunk ignore the already indexed data? OPSEC LEA should only be made to read the current file id to get the data which doesn't happen!

Please advise, is there any workaround ,

0 Karma

btiggemann
Path Finder

Ah yes,
that's what I have tried to tell you.
Just a short explanation of the inputs:

Audit = only everything audit related. Like admin user logon logoff, policy changes etc.
Firewall events = Everything that a firewall rule can log, for example eventlog for each rule, every time a session is opened
I am not sure, if this input also includes vpn events.
Smart defense = everything IPS related

Non-Audit = This includes every input that is NOT audit. So if you enable this input, this will collect Firewall Events and Smart Defense events.

That's why you get the duplicates.

To solve your problem please try to:
1. just enable Audit and Non-Audit and you will get every event only once

It is also possible to disable Non-Audit and keep Smart Defeinse and Firewall events enabled.
I am not sure if something is missing then.

Kieffer87
Communicator

We just enabled the Anti-Malware blade which required me to turn on Non-Audit. I couldn't for the life of me figure out why our ingestion rate doubled...This makes perfect sense, though I wish it was explained a bit better in the add-on documentation.

0 Karma

bbraun
New Member

Ill give this a shot. Thanks for the reply.

0 Karma

btiggemann
Path Finder

Hi,
we have updated to the most recent version of TA opsec and everything works fine now.
What kind of imports are you using? Be sure to just use non_audit and audit to catch all events. If you use the input vpn and non_audit you will index all vpn events twice

0 Karma

bbraun
New Member

We are on version 4.1.0 and have the following inputs configured: Firewall Audit, Firewall Events, Non-Audit, and SmartDefense.

When I run index=checkpoint for a one minute span I get the following distribution of sourcetypes:
opsec = 251,511
opsec:smartdefense = 3198
opsec:threat_emulation = 810
opsec:anti_malware = 144
opsec:anti_virus = 78

When I run index=checkpoint | dedup _raw for the same minute I get the following distribution of sourcetypes:
opsec = 154,489
opsec:smartdefense = 1599
opsec:threat_emulation = 810
opsec:anti_malware = 144
opsec:anti_virus = 78

This tells me that events for the first two sourcetypes are getting duplicated.

0 Karma

Kieffer87
Communicator

Non-Audit will collect SmartDefense and Firewall events...Remove those two inputs from your inputs and you should be good to go. In the end you should only need to have an input for Firewall Audit and Non-Audit.

0 Karma

N92
Path Finder

If I want to add two different types then I have to add two different inputs in OPSEC add on?

0 Karma

Kieffer87
Communicator

Generally if you want to collect all events except audit, you can just use the non-audit option. If you wanted to also collect audit, you would create a second input for audit.

0 Karma

N92
Path Finder

Thanks @Kiefffer87

For audit event: how it will differ from other events. It have different sourcetype? If yes then what would be name of it. If we are not getting it then how we can check?

0 Karma

Kieffer87
Communicator

http://docs.splunk.com/Documentation/AddOns/released/OPSEC-LEA/Sourcetypes

It will show up as a sourcetype of opsec:audit.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

Hi, this is a bug in the add-on; we're working on a fix.

0 Karma

rishrai
New Member

Hi Is this bug fixed? We have the same issue.

thanks,
Rish

0 Karma

bbraun
New Member

Hi Guys,

Any chance this bug is fixed? We are having the same issue!

0 Karma

btiggemann
Path Finder

Hi Splunkers, we have the same problem. Is there a workaround now? Is there a release date for the new version?
We are running in big license problems....

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...