CEF parsing using custom field labels and the CEFutils app

[khourihan_splun]

I have some CEF logs (Imperva) that I'd like to be able to parse and use custom field labels. I'd like to convert the custom label (cs1Label) to the field name, and the value to cs1. See below for an example:

Jun 13 21:09:06 10.241.3.230 CEF: 0|Imperva Inc.|SecureSphere|10.0.0|Protocol|Malformed JSON Message|Medium|act=None dhost=sn.api.np.km.playstation.net dpt=1084 duser=n/a src=90.148.63.132 spt=3686 proto=TCP end=Jun 13 2014 05:25:43 cat=SecurityEvent cs1=OrbisAZ-B cs1Label=ServerGroup cs2=Multiple cs2Label=ApplicationName cs3= cs3Label=QueryParameters cs4=Distributed Malformed JSON Message cs4Label=Description cs5=POST cs5Label=URLMethod cs6=kmj-us-p1-np-Nwafgw-3-222 cs6Label=Gateway destinationServiceName=orbisapi cn1=1728628311230694325 cn1Label=EventID cn2=62431 cn2Label=AlertNumber cn3=401 cn3Label=ResponseCode flexString1=181 flexString1Label=ResponseSize flexString2=16 flexString2Label=ResponseTime request=/socialnetwork/api/v1/users/me/opengraph


 

the app breaks down custom field labels as follows:

cs1=OrbisAZ-B cs1Label=ServerGroup 
cs2=Multiple cs2Label=ApplicationName 
cs3= cs3Label=QueryParameters 
cs4=Distributed Malformed JSON Message cs4Label=Description 
cs5=POST cs5Label=URLMethod 
cs6=kmj-us-p1-np-Nwafgw-3-222 cs6Label=Gateway destinationServiceName=orbisapi 
cn1=1728628311230694325 cn1Label=EventID 
cn2=62431 cn2Label=AlertNumber 
cn3=401 cn3Label=ResponseCode 
flexString1=181 flexString1Label=ResponseSize 
flexString2=16 flexString2Label=ResponseTime 

 
and within splunk, using the cefutils app, I would expect Splunk to see and offer KV pairs like:
 

cs1=OrbisAZ-B cs1Label=ServerGroup 
becomes
ServerGroup=OrbisAZ-B
 
cs2=Multiple cs2Label=ApplicationName 

becomes

ApplicationName=Multiple
 
cs4=Distributed Malformed JSON Message cs4Label=Description 

becomes

Description="Distributed Malformed JSON Message"

 
and, as explained in the cefutils readme file, a key name with a space turns into a _.
 
Excerpt below.

 

The application also provides a 'cefkv' command that should be used for extracting custom keys/value pairs from CEF data - useful if you are working with Arcsight.
Example:

CEF:0|Splunk|Test|1.0|signature:2|Test event|5|cs1=custom string value cs1Label=custom label
'cefkv' will extract following key/value pair from the sample message above:
custom_label="custom string value"

Needless to say this is not working, wondering if anyone out there has successfully done this?

Thanks,
Kyle

[aab5272]

I want to import logs from Imperva DAM ,can I install this add-on UF to format the data and send to indexer? Also what kind of configuration would be required for this?

thanks in advance

[IgorB · Accepted solution]

@khourihan_splunk - the reason the add-on is not working for you is because your data doesn't comply with CEF format as defined in this document: there's a space before CEF Version field: CEF: 0|Imperva <...snip...>.
| cefkv provides expected results once the space is removed:

AlertNumber         62431
ApplicationName     Multiple
Description         Distributed Malformed JSON Message
EventID             1728628311230694325
Gateway             kmj-us-p1-np-Nwafgw-3-222
QueryParameters      
ResponseCode        401
ServerGroup         OrbisAZ-B
URLMethod           POST 

So you've got 2 solutions here:
1. Have your source send events with properly formatted CEF headers
2. Modify the regex in line 11 of $SPLUNK_HOME/apps/cefutils/bin/cefkv.py to match your headers.

Sorry for the late answer. IMO that's still better than leaving the question unanswered ...

BR
--Igor

[aab5272]

@khourihan_splunk
I checked my logs it has the space as shown below
Jul 7 11:21:56 180.189.15.13 CEF: 0|Imperva Inc.|...

so I followed the steps mentioned above to modify the regex in cefkv.py at line 11 . here is my modification
CEF_DATA_RE = re.compile('CEF:\s\d+|[^|]|[^|]|[^|]|[^|]|[^|]|[^|]|(.*)')

I still don't get the expected key value pair can you recommend .

[aab5272]

@IgorB
I checked my logs it has the space as shown below
Jul 7 11:21:56 180.189.15.13 CEF: 0|Imperva Inc.|...

so I followed the steps you mentioned to modify the regex in cefkv.py at line 11 . here is my modification
CEF_DATA_RE = re.compile('CEF:\s\d+|[^|]|[^|]|[^|]|[^|]|[^|]|[^|]|(.*)')

I still don't get the expected key value pair can you recommend .

[reswob4]

Has @IgorB seen this?

[IgorB]

Apparently I missed this one. Thanks for pointing out

[kmcconnell]

Any chance you figured out how to extract the CEF key/value pairs? I'm in the same situation.

[dshpritz]

A few things:

1. have you tried my TA?

2. Wouldn't it be nice if Splunk had a KV_MODE = cef? (hint hint)