Splunk Search

regex and BREAK_ONLY_BEFORE

Branden
Builder

I have a script that sends something like the following to stdout:

DEV#:    0    DEVICE NAME: vpath0    TYPE: 2107900    POLICY: Optimized
SERIAL: 123bac
=======================================================================
Path#         Adapter/Hard Disk       State    Mode      Select  Errors
   0             fscsi0/hidsk22       Open     NORMAL    123456       0
   1             fscsi0/hidsk29       Open     NORMAL    456789       0

DEV#:    1    DEVICE NAME: vpath1    TYPE: 2107900    POLICY: Optimized
SERIAL: 123bac
=======================================================================
Path#         Adapter/Hard Disk       State    Mode      Select  Errors
   0             fscsi0/hidsk21       Open     NORMAL    123456       0
   1             fscsi0/hidsk28       Open     NORMAL    456789       0

etc... This list goes on for as many vpaths as there are on the host.

I'd like to capture this via Splunk, but I need to split it up by vpath entry; I don't want the entire output to be one big entry.

The script runs from $PLUNK_HOME/etc/apps/fc/bin/foo.sh on the forwarder and gets indexed as sourcetype "datapath-device" I added a props.conf on the forwarder in $SPLUNK_HOME/etc/apps/fc/local with the following entry:

[datapath-device]
BREAK_ONLY_BEFORE = ^DEV#:\s+\d+\s\w+\s\w+:\s\w\s+\w+:\s\w+\s+\w+:\w+

Unfortunately, it's not breaking up the output. Instead I get one huge entry with info on all the vpaths. I'm not sure if my regular expression is wrong or if props.conf is in the wrong place.

Any suggestions are appreciated.

Thanks!

Tags (1)
1 Solution

Brian_Osburn
Builder

You need to update the props.conf in $SPLUNK_HOME/etc/system/local on the indexer, not the forwarder..

I was able to use the following entry (replace TEST with what your sourcetype is)

[test]

BREAK_ONLY_BEFORE = ^DEV#:

Edit: Also, you may have to re-index the data to pick up the changes...

Brian

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

Yeah, as Brian Osburn says, this configuration must go where-ever the parsing phase occurs. See here.

I notice there are also no timestamps, so you might want to add a line:

DATETIME_CONFIG = CURRENT

(or some other more complicated config if you need to get it from the file name)

gkanapathy
Splunk Employee
Splunk Employee

You're not going to be able to do this very well with search-time regex. What you probably need is the "multikv" command. You might need to do a change on _raw (using eval _raw=replace(_raw,... and/or rex first) to clear out the header after extracting useful information from those. Actually it might be better to post this as another question, it needs fuller answer than I have room for here.

Branden
Builder

Actually, as a follow-up question to Brian and gkanapthy...
I have those two lines as part of the output:
0 fscsi0/hdisk1 OPEN NORMAL 123456 0
1 fscsi1/hdisk2 OPEN NORMAL 345667 0

I'd like to create a regular expression that pulls out the fields from the first line, then a regular expression to pull the fields from the second line (though the fields would have slightly different names from one line to another). In other words, I'd like a path_num0 and a path_num1 field. How do I use regex to distinguish one line from the next?
Thanks!

0 Karma

Brian_Osburn
Builder

You need to update the props.conf in $SPLUNK_HOME/etc/system/local on the indexer, not the forwarder..

I was able to use the following entry (replace TEST with what your sourcetype is)

[test]

BREAK_ONLY_BEFORE = ^DEV#:

Edit: Also, you may have to re-index the data to pick up the changes...

Brian

Branden
Builder

Worked beautifully, thanks!!!

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...