Developing for Splunk Enterprise

CEF logs parsing for Enterprise Security

ikulcsar
Communicator

Hi,

I know there are several questions with CEF in the subject, but I can not find a certain answer for my problem.
I have a Splunk deployment with Enterprise Security (and maybe with ITSI in the future). The goal is to analyze some logs received in CEF format. I already tried CEF Extraction Add-on for Splunk Enterprise, Support Add-on for ArcSight CEF inputs and Template for onboarding CEF data for CIM compliance, too.
My problem is the same with all Add-on: neither handle the custom labels/fields as I except:
cn2 = 4
cn2Label = TaskNewState
cs2 = 1093
cs2Label = ProductName
cs3 = 1.0.0.0
cs3Label = ProductVersion
etc...

But the goal:
TaskNewState = 4
ProductName = 1093
ProductVersion = 1.0.0.0

CEF Extraction Add-on has a command ( ... | cefkv ) which do the trick but doesn't seem to good for ES (, ITSI, ...).

So is there a way to parse the logs in this way/format (black magic is also accepted)? Any workaround?
What should I do?

Regards,
István

Tags (2)
0 Karma
1 Solution

dshpritz
SplunkTrust
SplunkTrust

The Template app is able to do this, provided you create your own TA using it as a base (hence the template part). Can you post some sample data?

Edit: Updated with info from the sample data

Looks like you found a bug in the template!

Here is what I have done to get the onboarding and extractions working correctly, using the CEF template add-on as a base:

Prepare your new add-on

I downloaded a copy of the add-on from splunkbase, then I renamed it to TA-kasperskylab_securitycenter.

Then I opened default/props.conf and changed the stanza from my_sourcetype to kasperskylab:securitycenter. Additionally, I renamed the REPORT extractions as such:

REPORT-cefLabelFirst = kasperskylab_cefLabelFirst
REPORT-cefLabelSecond = kasperskylab_cefLabelSecond
REPORT-builtInCefFields = kasperskylab_builtInCefFields

and opened the transforms.conf to match for those.

Configure onboarding

I configured the sourcetype for onboarding as such:

[kasperskylab:securitycenter:cef]
MAX_TIMESTAMP_LOOKAHEAD = 10
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %s
TIME_PREFIX = \|rt=
TRUNCATE=999999

Fix that extraction

I've uploaded a new version (1.2) to Splunkbase, but it may not be published yet. In the meantime, you can change the following line for the initial extractions from:

\sCEF:\d\|(?<vendor>[^\|]+)\|(?<product>[^\|]+)\|(?<product_version>[^\|]+)\|(?<signature_id>[^\|]+)\|(?<signature>[^\|]+)\|(?<vendor_severity>[^\|]+)\|(?<cef_message>.*)

to

^CEF:\d\|(?<vendor>[^\|]+)\|(?<product>[^\|]+)\|(?<product_version>[^\|]+)\|(?<signature_id>[^\|]+)\|(?<signature>[^\|]+)\|(?<vendor_severity>[^\|]+)\|(?<cef_message>.*)

CIM compliance

This fix should get the extractions going as you would like. Note that while this will get you a good start on the CIM fields, it isn't a silver bullet. You will still need to add eventtypes and tags, and may need to add some field aliases or extractions to round out the model with which you decide to work.

HTH,

Dave

Edit: Here's a screenshot of the results:
alt text

View solution in original post

micahkemp
Champion

Now with ProductName=1093 goodness:

transforms.conf:

[kasperskycefcomponents]
REGEX = ^CEF:(?<cef_cefVersion>[^|]+)\|(?<cef_vendor>[^|]+)\|(?<cef_product>[^|]+)\|(?<cef_version>[^|]+)\|(?<cef_signature>[^|]+)\|(?<cef_name>[^|]+)\|(?<cef_severity>[^|]+)\|(?<ceffields>.*)$

[kasperskyceffields]
SOURCE_KEY = ceffields
REGEX = (?<_KEY_1>[^ |=]+)=(?<_VAL_1>[^=]+)( |$)

[kasperskeyceffields_combined]
SOURCE_KEY = ceffields
REGEX = ([^ |=]+)=(?<_VAL_1>[^=]+) (\1Label)=(?<_KEY_1>[^=]+)( |$)

props.conf:

[<sourcetype>]
SHOULD_LINEMERGE = false
KV_MODE = none
REPORT-kasperskycef = kasperskycefcomponents,kasperskyceffields,kasperskeyceffields_combined

In action:
alt text

And with random spaces thrown into the values/Labels:
alt text

0 Karma

micahkemp
Champion

This will parse into the fields you've mentioned (assuming key=value), but does nothing to make them CIM compliant:

transforms.conf:

[kasperskycefcomponents]
REGEX = ^CEF:(?<cef_cefVersion>[^|]+)\|(?<cef_vendor>[^|]+)\|(?<cef_product>[^|]+)\|(?<cef_version>[^|]+)\|(?<cef_signature>[^|]+)\|(?<cef_name>[^|]+)\|(?<cef_severity>[^|]+)\|(?<ceffields>.*)$

[kasperskyceffields]
SOURCE_KEY = ceffields
REGEX = (\t|^)(?<_KEY_1>[^=]+)=(?<_VAL_1>[^\t]+)(\t|$)

props.conf:

[<sourcetype name>]
SHOULD_LINEMERGE = false
KV_MODE = none
REPORT-kasperskycef = kasperskycefcomponents,kasperskyceffields

There are better ways to do this if you know that you care about specific fields, but the above is a general solution to the CEF you posted.

If I was incorrect about there being tabs between key=value pairs, transforms.conf would look like this instead:

[kasperskycefcomponents]
REGEX = ^CEF:(?<cef_cefVersion>[^|]+)\|(?<cef_vendor>[^|]+)\|(?<cef_product>[^|]+)\|(?<cef_version>[^|]+)\|(?<cef_signature>[^|]+)\|(?<cef_name>[^|]+)\|(?<cef_severity>[^|]+)\|(?<ceffields>.*)$

[kasperskyceffields]
SOURCE_KEY = ceffields
REGEX = (?<_KEY_1>[^ =]+)=(?<_VAL_1>[^=]+)( |$) 
0 Karma

ikulcsar
Communicator

Thank you for your answer. Unfortunately, it doesn't handle correctly the custom labels, and field values with white spaces. Sorry for the badly formatted question, I will check the guide.

Regards,
Istvan

0 Karma

micahkemp
Champion

Despite @dspritz already having posted the better (and more definitive) answer, I'm wondering if you could detail what my answer didn't correctly handle, so that I can refine my own abilities.

0 Karma

ikulcsar
Communicator

Hi there,

After the search time extraction the result looks like this, at least for me:
cs2 = 1093
cs2Label = ProductName
cs3 = 1.X.X.X
cs3Label =ProductVersion
dhost = MXXXXXXX
dst = 127.0.0.1

And my expectations look like something this:
ProductName = 1093
ProductVersion = 1.X.X.X
dhost = MXXXXXXX
dst = 127.0.0.1

Regards,
Istvan

0 Karma

micahkemp
Champion

Ahhhh, I misunderstood your requirements. Thanks for taking the time to reply!

0 Karma

ikulcsar
Communicator

Welcome. Thank you for your help and time.

0 Karma

ikulcsar
Communicator

Thank you for all for the reply. Since all of you have asked for logs, I will reply in a new thread.
Here 4 example logs line. For the first one, I copied the parsed part too. As far as I've seen, every log line parsing behaves the same way. For this, I used CEF Extraction Add-on on (almost) default config.

CEF:0|KasperskyLab|SecurityCenter|10.X.X|KLSRV_HOST_STATUS_WARNING|Device status is Warning|2|msg=Status of device 'NXXXXX' changed to Warning: KSN servers unavailable. rt=1514445830 dhost=MXXXXXXX dst=127.0.0.1 cs2=1093 cs2Label=ProductName cs3=1.X.X.X cs3Label=ProductVersion

cef_cefVersion = 0

cef_name = Device status is Warning
cef_product = SecurityCenter

cef_severity = 2

cef_signature = KLSRV_HOST_STATUS_WARNING

cef_vendor = KasperskyLab

cef_version = 10.X.X
cs2 = 1093

cs2Label = ProductName

cs3 = 1.X.X.X
cs3Label = ProductVersion

dhost = MXXXXXXX

dst = 127.0.0.1
msg = Status of device 'NXXXXX' changed to Warning: KSN servers unavailable.

rt = 1514445830

CEF:0|KasperskyLab|SecurityCenter|10.X.X|KLNAG_EV_INV_APP_UNINSTALLED|Application has been removed|1|msg='Intel® PROSet/Wireless Software' version '19.1.0' has been removed rt=1514445754 dhost=NXXXXX dst=10.X.X.X cs2=1103 cs2Label=ProductName cs3=1.X.X.X cs3Label=ProductVersion fname=Intel® PROSet/Wireless Software fileType=application

CEF:0|KasperskyLab|SecurityCenter|10.X.X|KLPRCI_TaskState|Running|1|rt=1514445608 dhost=NXXXXX dst=1X.X.X.X cs2=KES cs2Label=ProductName cs3=10.X.X.X cs3Label=ProductVersion cs5=Install update cs5Label=TaskName cs4=102 cs4Label=TaskId cn2=1 cn2Label=TaskNewState cn1=0 cn1Label=TaskOldState

CEF:0|KasperskyLab|SecurityCenter|10.X.X|KLSRV_INVISIBLE_HOSTS_REMOVED|Invisible device removed automatically|1|msg=Device "NXXXXX" was automatically removed from group "Managed devices/Notebooks/" as it had not been visible for 62 days. rt=1514445073 dhost=MXXXXXXX dst=127.0.0.1 cs2=1093 cs2Label=ProductName cs3=1.X.X.X cs3Label=ProductVersion fname=Managed devices/Notebooks/ fileType=administrationGroup

Please note that CIM and parsing area are new to me.
Regards,
István

0 Karma

micahkemp
Champion

Please format your sample data using the code button (shown as 101010). My post assumed there are tabs, but others believe that to be incorrect. It would be easier if it was definitively shown in your post.

0 Karma

dshpritz
SplunkTrust
SplunkTrust

The Template app is able to do this, provided you create your own TA using it as a base (hence the template part). Can you post some sample data?

Edit: Updated with info from the sample data

Looks like you found a bug in the template!

Here is what I have done to get the onboarding and extractions working correctly, using the CEF template add-on as a base:

Prepare your new add-on

I downloaded a copy of the add-on from splunkbase, then I renamed it to TA-kasperskylab_securitycenter.

Then I opened default/props.conf and changed the stanza from my_sourcetype to kasperskylab:securitycenter. Additionally, I renamed the REPORT extractions as such:

REPORT-cefLabelFirst = kasperskylab_cefLabelFirst
REPORT-cefLabelSecond = kasperskylab_cefLabelSecond
REPORT-builtInCefFields = kasperskylab_builtInCefFields

and opened the transforms.conf to match for those.

Configure onboarding

I configured the sourcetype for onboarding as such:

[kasperskylab:securitycenter:cef]
MAX_TIMESTAMP_LOOKAHEAD = 10
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %s
TIME_PREFIX = \|rt=
TRUNCATE=999999

Fix that extraction

I've uploaded a new version (1.2) to Splunkbase, but it may not be published yet. In the meantime, you can change the following line for the initial extractions from:

\sCEF:\d\|(?<vendor>[^\|]+)\|(?<product>[^\|]+)\|(?<product_version>[^\|]+)\|(?<signature_id>[^\|]+)\|(?<signature>[^\|]+)\|(?<vendor_severity>[^\|]+)\|(?<cef_message>.*)

to

^CEF:\d\|(?<vendor>[^\|]+)\|(?<product>[^\|]+)\|(?<product_version>[^\|]+)\|(?<signature_id>[^\|]+)\|(?<signature>[^\|]+)\|(?<vendor_severity>[^\|]+)\|(?<cef_message>.*)

CIM compliance

This fix should get the extractions going as you would like. Note that while this will get you a good start on the CIM fields, it isn't a silver bullet. You will still need to add eventtypes and tags, and may need to add some field aliases or extractions to round out the model with which you decide to work.

HTH,

Dave

Edit: Here's a screenshot of the results:
alt text

View solution in original post

ikulcsar
Communicator

Thank you very much, it looks good.
I will check the other questions/answers for guides/best practices how to set event types/tags. (What event types should I define, etc.)

Regards,
Istvan

0 Karma

gwalford
Path Finder

It has been over 2 years since I last looked at CEF, and if my memory serves me correctly, I had to download one of the CEF applications off of Splunkbase, then modify the parsing slightly to conform with Arcsight. It did not take too much work to do.

GIven that you are dealing with key=value pairs, I would suggest you use a field extraction to get the details you want.

0 Karma

micahkemp
Champion

Can you include some of the log lines that aren't being parsed as you want?

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!