Splunk Search

Help extracting multiline events using rex

jambajuice
Communicator

We have events that look like this:

edit 4
    set srcintf "port1"
    set dstintf "port2"
        set srcaddr "0.0.0.0" "127.0.0.0" "169.254.0.0" "172.16.0.0" "192.0.2.0" "192.168.0.0" "255.255.255.255"             
        set dstaddr "ANY"             
    set schedule "always"
        set service "ANY"             
    set logtraffic enable
    set comments "Blocks inbound traffic from illegal networks"
next
edit 56
    set srcintf "port1"
    set dstintf "port1"
        set srcaddr "0.0.0.0" "127.0.0.0" "169.254.0.0" "172.16.0.0" "192.0.2.0" "192.168.0.0" "255.255.255.255"             
        set dstaddr "ANY"             
    set schedule "always"
        set service "ANY"             
    set logtraffic enable
next
edit 1
    set srcintf "port1"
    set dstintf "port2"
        set srcaddr "ANY"             
        set dstaddr "jnetwebcluster_VIP"             
    set action accept
    set utm-status enable
    set comments "Permit inbound ICMP to JNET DMZ"
    set schedule "always"
        set service "Allowed-ICMP"             
    set av-profile "strict"
    set ips-sensor "all_default_pass"
    set dlp-sensor "Credit-Card"
    set profile-protocol-options "strict"
    set logtraffic enable
next

I am trying to extract each individual rule as a separate event using rex. I've tried the following:

rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*next)"  
rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*)next"  
rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*)(?=next)"  
rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*)\n\s+next"  

Each one of these extractions will put all of the individual rules into a single field. It won't terminate at the "next" line that ends each individual rule.

What am I doing wrong here?

Thx.

Tags (1)

gkanapathy
Splunk Employee
Splunk Employee

For your current data, you need to use a non-greedy regex match:

(?msi)edit\s(?P<rule>.*?)\vnext

is probably what you need. Note I use .*? instead of .*.


But it seems that you would probably be a lot better off breaking this at index time with:

[mysourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = (?i)([\r\n]+)(?=edit)

Or (less efficiently but maybe easier to understand):

[mysourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = (?i)^edit

I feel like I've answered this question before.

Ron_Naken
Splunk Employee
Splunk Employee

From what I can gather, you would be best off adding an entry to props.conf to allow Splunk to recognize each entry as a single event. You could do that like this:

$SPLUNK_HOME/etc/system/local/PROPS.CONF:

[mysourcetype]
SHOULD_LINEMERGE = true
MUST_BREAK_AFTER =^next$

This won't affect existing data, but it will allow Splunk to recognize each edit clause as a single entry for new data.

gkanapathy
Splunk Employee
Splunk Employee

much better would be to use SHOULD_LINEMERGE = false and LINE_BREAKER. See my response.

0 Karma

Ayn
Legend

Is there a specific reason for wanting to use rex for this? Otherwise you have transaction which is made just for grouping events together like this.

<yoursearch> | transaction startswith="edit" endswith="next"

More information on transaction: http://www.splunk.com/base/Documentation/latest/SearchReference/Transaction

0 Karma

jambajuice
Communicator

I thought transaction was for grouping multiple events together, not for expanding a single event into multiple fields/events.

I tried using your example and it returns no results. The problem is that there are a number of nested edit/next statements within each policy. I've got the event breaking configured to break the config file into events where the edit/next lines are at the start of the line. This creates a single event for each major config section. I need to break one of those sections into individual events.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...