topic Re: regex and BREAK_ONLY_BEFORE in Splunk Search

regex and BREAK_ONLY_BEFORE

Branden — Tue, 31 Aug 2010 00:12:40 GMT

I have a script that sends something like the following to stdout:

DEV#:    0    DEVICE NAME: vpath0    TYPE: 2107900    POLICY: Optimized
SERIAL: 123bac
=======================================================================
Path#         Adapter/Hard Disk       State    Mode      Select  Errors
   0             fscsi0/hidsk22       Open     NORMAL    123456       0
   1             fscsi0/hidsk29       Open     NORMAL    456789       0

DEV#:    1    DEVICE NAME: vpath1    TYPE: 2107900    POLICY: Optimized
SERIAL: 123bac
=======================================================================
Path#         Adapter/Hard Disk       State    Mode      Select  Errors
   0             fscsi0/hidsk21       Open     NORMAL    123456       0
   1             fscsi0/hidsk28       Open     NORMAL    456789       0

etc... This list goes on for as many vpaths as there are on the host.

I'd like to capture this via Splunk, but I need to split it up by vpath entry; I don't want the entire output to be one big entry.

The script runs from $PLUNK_HOME/etc/apps/fc/bin/foo.sh on the forwarder and gets indexed as sourcetype "datapath-device" I added a props.conf on the forwarder in $SPLUNK_HOME/etc/apps/fc/local with the following entry:

[datapath-device]
BREAK_ONLY_BEFORE = ^DEV#:\s+\d+\s\w+\s\w+:\s\w\s+\w+:\s\w+\s+\w+:\w+

Unfortunately, it's not breaking up the output. Instead I get one huge entry with info on all the vpaths. I'm not sure if my regular expression is wrong or if props.conf is in the wrong place.

Any suggestions are appreciated.

Thanks!

Re: regex and BREAK_ONLY_BEFORE

Brian_Osburn — Tue, 31 Aug 2010 00:23:00 GMT

You need to update the props.conf in $SPLUNK_HOME/etc/system/local on the indexer, not the forwarder..

I was able to use the following entry (replace TEST with what your sourcetype is)

[test]

BREAK_ONLY_BEFORE = ^DEV#:

Edit: Also, you may have to re-index the data to pick up the changes...

Brian

Re: regex and BREAK_ONLY_BEFORE

Branden — Tue, 31 Aug 2010 00:35:46 GMT

Worked beautifully, thanks!!!

Re: regex and BREAK_ONLY_BEFORE

gkanapathy — Tue, 31 Aug 2010 00:39:23 GMT

Yeah, as Brian Osburn says, this configuration must go where-ever the parsing phase occurs. See here.

I notice there are also no timestamps, so you might want to add a line:

DATETIME_CONFIG = CURRENT

(or some other more complicated config if you need to get it from the file name)

Re: regex and BREAK_ONLY_BEFORE

Branden — Mon, 28 Sep 2020 09:16:52 GMT

Actually, as a follow-up question to Brian and gkanapthy...
I have those two lines as part of the output:
0 fscsi0/hdisk1 OPEN NORMAL 123456 0
1 fscsi1/hdisk2 OPEN NORMAL 345667 0

I'd like to create a regular expression that pulls out the fields from the first line, then a regular expression to pull the fields from the second line (though the fields would have slightly different names from one line to another). In other words, I'd like a path_num0 and a path_num1 field. How do I use regex to distinguish one line from the next?
Thanks!

Re: regex and BREAK_ONLY_BEFORE

gkanapathy — Mon, 28 Sep 2020 09:16:55 GMT

You're not going to be able to do this very well with search-time regex. What you probably need is the "multikv" command. You might need to do a change on _raw (using eval _raw=replace(_raw,... and/or rex first) to clear out the header after extracting useful information from those. Actually it might be better to post this as another question, it needs fuller answer than I have room for here.