Getting Data In

How to configure Splunk preveing parsing multiline Imperva logs as separate events?

charanyack16
New Member

Imperva to Splunk - Unable to properly parse multiline events. Rawquery fields are appended with different timestamps for each newline.

EX:

Event 1

Jul 11 09:18:18 abc.xyz.com CEF: 0|Imperva Inc.|SecureSphere|10.5.0.13_0|XYZ|XYZ.DAM|Informative|dest_ip=qq.tyy.214.28 dest_port=0000 dbuser=xyz sou_ip=00.000.000.00 source=000 proto=TCP createtime=09 July 2000 16:18:10, cat=XYZ Configuration Changes servergroup=XY MSServer servicename =YZ QLServer applicationname=XYZ App event_id=00000 query=Query usergroup=Default autheticated=Yes applicationuser= application= osuser= hostname= dbname=xyzmanager schemaname= bindvariable= errorvalue= responsesize=0 responsetime=0 affectedrows=0 parsequery=if object_id(?) is not ? drop table #entitytransaction create table #entitytransaction (transactionid bigint) rawdata=#015

Event 2

Jul 11 09:18:18 abc.xyz.com                     CREATE TABLE #EntityTrans (TransId bigint)#015

Ideally they are a single event, but Splunk displays them as two different events

0 Karma

hardikJsheth
Motivator

You need to configure two attributes (SHOULD_LINEMERGE and BREAK_ONLY_BEFORE) in your props.conf file for your source type.

For e.g

[imperva]
SHOULD_LINEMERGE =true
BREAK_ONLY_BEFORE = abc.xyz.com

Update source type and BREAK_ONLY_BEFORE to appropriate value as per your requirement. Also note, this configurations are required on indexer node in clustered environment.

0 Karma

charanyack16
New Member

The props.conf file is shown below. The last field name in the log (refer to the log above) "parsequery" has the value,
parsequery=if object_id(?) is not ? drop table #entitytransaction create table #entitytransaction (transactionid bigint) rawdata=#015----------space-------------- CREATE TABLE #EntityTrans (TransId bigint)#015

This is different from rest of the field values.You see lot of space between "rawdata=#015----------------space------------------------- CREATE TABLE #EntityTrans (TransId bigint)#015"

How do I configure attribute SHOULD_LINEMERGE and BREAK_ONLY_BEFORE ?

[imperva:dam:uswc]
SHOULD_LINEMERGE = true
NO_BINARY_CHECK=true
CHARSET=UTF-8
KV_MODE=none
disabled=false
pulldown_type=true
TZ=America/Los_Angeles
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 15
REPORT-0basic_fields_uswc = extract_basic_info_uswc
REPORT-1app_fields_uswc = extract_dam_fields_uswc
BREAK_ONLY_BEFORE =\s+CEF
0 Karma

hardikJsheth
Motivator

You can have following parameters.

SHOULD_LINEMERGE = true
MUST_BREAK_AFTER =(parsequery=.*)abc.xyz.com
BREAK_ONLY_BEFORE =(abc.xyz.com\sCEF:)

Please replace abc.xyz.com with appropriate host name as you are getting in your logs. Also ensure this configurations are implemented on indexer node in case of distributed architecture.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...