Imperva to Splunk - Unable to properly parse multiline events. Rawquery fields are appended with different timestamps for each newline.
EX:
Event 1
Jul 11 09:18:18 abc.xyz.com CEF: 0|Imperva Inc.|SecureSphere|10.5.0.13_0|XYZ|XYZ.DAM|Informative|dest_ip=qq.tyy.214.28 dest_port=0000 dbuser=xyz sou_ip=00.000.000.00 source=000 proto=TCP createtime=09 July 2000 16:18:10, cat=XYZ Configuration Changes servergroup=XY MSServer servicename =YZ QLServer applicationname=XYZ App event_id=00000 query=Query usergroup=Default autheticated=Yes applicationuser= application= osuser= hostname= dbname=xyzmanager schemaname= bindvariable= errorvalue= responsesize=0 responsetime=0 affectedrows=0 parsequery=if object_id(?) is not ? drop table #entitytransaction create table #entitytransaction (transactionid bigint) rawdata=#015
Event 2
Jul 11 09:18:18 abc.xyz.com CREATE TABLE #EntityTrans (TransId bigint)#015
Ideally they are a single event, but Splunk displays them as two different events
You need to configure two attributes (SHOULD_LINEMERGE and BREAK_ONLY_BEFORE) in your props.conf file for your source type.
For e.g
[imperva]
SHOULD_LINEMERGE =true
BREAK_ONLY_BEFORE = abc.xyz.com
Update source type and BREAK_ONLY_BEFORE to appropriate value as per your requirement. Also note, this configurations are required on indexer node in clustered environment.
The props.conf file is shown below. The last field name in the log (refer to the log above) "parsequery" has the value,
parsequery=if object_id(?) is not ? drop table #entitytransaction create table #entitytransaction (transactionid bigint) rawdata=#015----------space-------------- CREATE TABLE #EntityTrans (TransId bigint)#015
This is different from rest of the field values.You see lot of space between "rawdata=#015----------------space------------------------- CREATE TABLE #EntityTrans (TransId bigint)#015"
How do I configure attribute SHOULD_LINEMERGE and BREAK_ONLY_BEFORE ?
[imperva:dam:uswc]
SHOULD_LINEMERGE = true
NO_BINARY_CHECK=true
CHARSET=UTF-8
KV_MODE=none
disabled=false
pulldown_type=true
TZ=America/Los_Angeles
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 15
REPORT-0basic_fields_uswc = extract_basic_info_uswc
REPORT-1app_fields_uswc = extract_dam_fields_uswc
BREAK_ONLY_BEFORE =\s+CEF
You can have following parameters.
SHOULD_LINEMERGE = true
MUST_BREAK_AFTER =(parsequery=.*)abc.xyz.com
BREAK_ONLY_BEFORE =(abc.xyz.com\sCEF:)
Please replace abc.xyz.com with appropriate host name as you are getting in your logs. Also ensure this configurations are implemented on indexer node in case of distributed architecture.