Help extracting multiline events using rex

[jambajuice]

We have events that look like this:

edit 4
    set srcintf "port1"
    set dstintf "port2"
        set srcaddr "0.0.0.0" "127.0.0.0" "169.254.0.0" "172.16.0.0" "192.0.2.0" "192.168.0.0" "255.255.255.255"             
        set dstaddr "ANY"             
    set schedule "always"
        set service "ANY"             
    set logtraffic enable
    set comments "Blocks inbound traffic from illegal networks"
next
edit 56
    set srcintf "port1"
    set dstintf "port1"
        set srcaddr "0.0.0.0" "127.0.0.0" "169.254.0.0" "172.16.0.0" "192.0.2.0" "192.168.0.0" "255.255.255.255"             
        set dstaddr "ANY"             
    set schedule "always"
        set service "ANY"             
    set logtraffic enable
next
edit 1
    set srcintf "port1"
    set dstintf "port2"
        set srcaddr "ANY"             
        set dstaddr "jnetwebcluster_VIP"             
    set action accept
    set utm-status enable
    set comments "Permit inbound ICMP to JNET DMZ"
    set schedule "always"
        set service "Allowed-ICMP"             
    set av-profile "strict"
    set ips-sensor "all_default_pass"
    set dlp-sensor "Credit-Card"
    set profile-protocol-options "strict"
    set logtraffic enable
next

I am trying to extract each individual rule as a separate event using rex. I've tried the following:

rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*next)"  
rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*)next"  
rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*)(?=next)"  
rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*)\n\s+next"  

Each one of these extractions will put all of the individual rules into a single field. It won't terminate at the "next" line that ends each individual rule.

What am I doing wrong here?

Thx.

[gkanapathy]

For your current data, you need to use a non-greedy regex match:

(?msi)edit\s(?P<rule>.*?)\vnext

is probably what you need. Note I use .*? instead of .*.

---

But it seems that you would probably be a lot better off breaking this at index time with:

[mysourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = (?i)([\r\n]+)(?=edit)

Or (less efficiently but maybe easier to understand):

[mysourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = (?i)^edit

I feel like I've answered this question before.

[Ron_Naken]

From what I can gather, you would be best off adding an entry to props.conf to allow Splunk to recognize each entry as a single event. You could do that like this:

$SPLUNK_HOME/etc/system/local/PROPS.CONF:

[mysourcetype]
SHOULD_LINEMERGE = true
MUST_BREAK_AFTER =^next$

This won't affect existing data, but it will allow Splunk to recognize each edit clause as a single entry for new data.

[gkanapathy]

much better would be to use SHOULD_LINEMERGE = false and LINE_BREAKER. See my response.

[Ayn]

Is there a specific reason for wanting to use rex for this? Otherwise you have transaction which is made just for grouping events together like this.

<yoursearch> | transaction startswith="edit" endswith="next"

More information on transaction: http://www.splunk.com/base/Documentation/latest/SearchReference/Transaction

[jambajuice]

I thought transaction was for grouping multiple events together, not for expanding a single event into multiple fields/events.

I tried using your example and it returns no results. The problem is that there are a number of nested edit/next statements within each policy. I've got the event breaking configured to break the config file into events where the edit/next lines are at the start of the line. This creates a single event for each major config section. I need to break one of those sections into individual events.