I have some CEF logs (Imperva) that I'd like to be able to parse and use custom field labels. I'd like to convert the custom label (cs1Label) to the field name, and the value to cs1. See below for an example:
Jun 13 21:09:06 10.241.3.230 CEF: 0|Imperva Inc.|SecureSphere|10.0.0|Protocol|Malformed JSON Message|Medium|act=None dhost=sn.api.np.km.playstation.net dpt=1084 duser=n/a src=90.148.63.132 spt=3686 proto=TCP end=Jun 13 2014 05:25:43 cat=SecurityEvent cs1=OrbisAZ-B cs1Label=ServerGroup cs2=Multiple cs2Label=ApplicationName cs3= cs3Label=QueryParameters cs4=Distributed Malformed JSON Message cs4Label=Description cs5=POST cs5Label=URLMethod cs6=kmj-us-p1-np-Nwafgw-3-222 cs6Label=Gateway destinationServiceName=orbisapi cn1=1728628311230694325 cn1Label=EventID cn2=62431 cn2Label=AlertNumber cn3=401 cn3Label=ResponseCode flexString1=181 flexString1Label=ResponseSize flexString2=16 flexString2Label=ResponseTime request=/socialnetwork/api/v1/users/me/opengraph
the app breaks down custom field labels as follows:
cs1=OrbisAZ-B cs1Label=ServerGroup
cs2=Multiple cs2Label=ApplicationName
cs3= cs3Label=QueryParameters
cs4=Distributed Malformed JSON Message cs4Label=Description
cs5=POST cs5Label=URLMethod
cs6=kmj-us-p1-np-Nwafgw-3-222 cs6Label=Gateway destinationServiceName=orbisapi
cn1=1728628311230694325 cn1Label=EventID
cn2=62431 cn2Label=AlertNumber
cn3=401 cn3Label=ResponseCode
flexString1=181 flexString1Label=ResponseSize
flexString2=16 flexString2Label=ResponseTime
and within splunk, using the cefutils app, I would expect Splunk to see and offer KV pairs like:
cs1=OrbisAZ-B cs1Label=ServerGroup
becomes
ServerGroup=OrbisAZ-B
cs2=Multiple cs2Label=ApplicationName
becomes
ApplicationName=Multiple
cs4=Distributed Malformed JSON Message cs4Label=Description
becomes
Description="Distributed Malformed JSON Message"
and, as explained in the cefutils readme file, a key name with a space turns into a _.
Excerpt below.
The application also provides a 'cefkv' command that should be used for extracting custom keys/value pairs from CEF data - useful if you are working with Arcsight.
Example:
CEF:0|Splunk|Test|1.0|signature:2|Test event|5|cs1=custom string value cs1Label=custom label
'cefkv' will extract following key/value pair from the sample message above:
custom_label="custom string value"
Needless to say this is not working, wondering if anyone out there has successfully done this?
Thanks,
Kyle
@khourihan_splunk - the reason the add-on is not working for you is because your data doesn't comply with CEF format as defined in this document: there's a space before CEF Version field: CEF: 0|Imperva <...snip...>
.
| cefkv
provides expected results once the space is removed:
AlertNumber 62431
ApplicationName Multiple
Description Distributed Malformed JSON Message
EventID 1728628311230694325
Gateway kmj-us-p1-np-Nwafgw-3-222
QueryParameters
ResponseCode 401
ServerGroup OrbisAZ-B
URLMethod POST
So you've got 2 solutions here:
1. Have your source send events with properly formatted CEF headers
2. Modify the regex in line 11 of $SPLUNK_HOME/apps/cefutils/bin/cefkv.py
to match your headers.
Sorry for the late answer. IMO that's still better than leaving the question unanswered ...
BR
--Igor
I want to import logs from Imperva DAM ,can I install this add-on UF to format the data and send to indexer? Also what kind of configuration would be required for this?
thanks in advance
@khourihan_splunk - the reason the add-on is not working for you is because your data doesn't comply with CEF format as defined in this document: there's a space before CEF Version field: CEF: 0|Imperva <...snip...>
.
| cefkv
provides expected results once the space is removed:
AlertNumber 62431
ApplicationName Multiple
Description Distributed Malformed JSON Message
EventID 1728628311230694325
Gateway kmj-us-p1-np-Nwafgw-3-222
QueryParameters
ResponseCode 401
ServerGroup OrbisAZ-B
URLMethod POST
So you've got 2 solutions here:
1. Have your source send events with properly formatted CEF headers
2. Modify the regex in line 11 of $SPLUNK_HOME/apps/cefutils/bin/cefkv.py
to match your headers.
Sorry for the late answer. IMO that's still better than leaving the question unanswered ...
BR
--Igor
@khourihan_splunk
I checked my logs it has the space as shown below
Jul 7 11:21:56 180.189.15.13 CEF: 0|Imperva Inc.|...
so I followed the steps mentioned above to modify the regex in cefkv.py at line 11 . here is my modification
CEF_DATA_RE = re.compile('CEF:\s\d+|[^|]|[^|]|[^|]|[^|]|[^|]|[^|]|(.*)')
I still don't get the expected key value pair can you recommend .
@IgorB
I checked my logs it has the space as shown below
Jul 7 11:21:56 180.189.15.13 CEF: 0|Imperva Inc.|...
so I followed the steps you mentioned to modify the regex in cefkv.py at line 11 . here is my modification
CEF_DATA_RE = re.compile('CEF:\s\d+|[^|]|[^|]|[^|]|[^|]|[^|]|[^|]|(.*)')
I still don't get the expected key value pair can you recommend .
Has @IgorB seen this?
Apparently I missed this one. Thanks for pointing out
Any chance you figured out how to extract the CEF key/value pairs? I'm in the same situation.
A few things:
have you tried my TA?
Wouldn't it be nice if Splunk had a KV_MODE = cef? (hint hint)