Splunk Search

Help extracting multiline events using rex

jambajuice
Communicator

We have events that look like this:

edit 4
    set srcintf "port1"
    set dstintf "port2"
        set srcaddr "0.0.0.0" "127.0.0.0" "169.254.0.0" "172.16.0.0" "192.0.2.0" "192.168.0.0" "255.255.255.255"             
        set dstaddr "ANY"             
    set schedule "always"
        set service "ANY"             
    set logtraffic enable
    set comments "Blocks inbound traffic from illegal networks"
next
edit 56
    set srcintf "port1"
    set dstintf "port1"
        set srcaddr "0.0.0.0" "127.0.0.0" "169.254.0.0" "172.16.0.0" "192.0.2.0" "192.168.0.0" "255.255.255.255"             
        set dstaddr "ANY"             
    set schedule "always"
        set service "ANY"             
    set logtraffic enable
next
edit 1
    set srcintf "port1"
    set dstintf "port2"
        set srcaddr "ANY"             
        set dstaddr "jnetwebcluster_VIP"             
    set action accept
    set utm-status enable
    set comments "Permit inbound ICMP to JNET DMZ"
    set schedule "always"
        set service "Allowed-ICMP"             
    set av-profile "strict"
    set ips-sensor "all_default_pass"
    set dlp-sensor "Credit-Card"
    set profile-protocol-options "strict"
    set logtraffic enable
next

I am trying to extract each individual rule as a separate event using rex. I've tried the following:

rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*next)"  
rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*)next"  
rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*)(?=next)"  
rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*)\n\s+next"  

Each one of these extractions will put all of the individual rules into a single field. It won't terminate at the "next" line that ends each individual rule.

What am I doing wrong here?

Thx.

Tags (1)

gkanapathy
Splunk Employee
Splunk Employee

For your current data, you need to use a non-greedy regex match:

(?msi)edit\s(?P<rule>.*?)\vnext

is probably what you need. Note I use .*? instead of .*.


But it seems that you would probably be a lot better off breaking this at index time with:

[mysourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = (?i)([\r\n]+)(?=edit)

Or (less efficiently but maybe easier to understand):

[mysourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = (?i)^edit

I feel like I've answered this question before.

Ron_Naken
Splunk Employee
Splunk Employee

From what I can gather, you would be best off adding an entry to props.conf to allow Splunk to recognize each entry as a single event. You could do that like this:

$SPLUNK_HOME/etc/system/local/PROPS.CONF:

[mysourcetype]
SHOULD_LINEMERGE = true
MUST_BREAK_AFTER =^next$

This won't affect existing data, but it will allow Splunk to recognize each edit clause as a single entry for new data.

gkanapathy
Splunk Employee
Splunk Employee

much better would be to use SHOULD_LINEMERGE = false and LINE_BREAKER. See my response.

0 Karma

Ayn
Legend

Is there a specific reason for wanting to use rex for this? Otherwise you have transaction which is made just for grouping events together like this.

<yoursearch> | transaction startswith="edit" endswith="next"

More information on transaction: http://www.splunk.com/base/Documentation/latest/SearchReference/Transaction

0 Karma

jambajuice
Communicator

I thought transaction was for grouping multiple events together, not for expanding a single event into multiple fields/events.

I tried using your example and it returns no results. The problem is that there are a number of nested edit/next statements within each policy. I've got the event breaking configured to break the config file into events where the edit/next lines are at the start of the line. This creates a single event for each major config section. I need to break one of those sections into individual events.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...