Out of the box, the unix sed
command operates on a line-by-line basis. Is this the same for the SEDCMD
setting in props.conf
? Or does it operate on the entire event?
Just for the sake of an example. Say I'm using the following unix command to filter out comment lines (lines starting with a ";"). (Technically, this example only clears the comment lines rather than remove them, "grep" would be more obvious choice of unix commands, but splunk doesn't have a GREPCMD
, so it's a mute point.)
cat myfile | sed -re 's/^\s*;.*$//'
Will that work the same way as the following props.conf
entry:
SEDCMD-drop_comments = s/^\*;.*$//
To get the SEDCMD
to work on a line-by-line basis, you can use the following:
SEDCMD-drop_comments = s/(?m)^\*;.*$//g
The difference is:
(?m)
enables multi-line mode in regex engine. This causes ^
and $
to now mean start-of-line and end-of-line, rather than start-of-event and end-of-event.g
on the end tells the sed command to be applied as many times as possible; therefore the replacement will occur more than once. So without the g
, only the first matching line would be replaced instead of all the lines.@gkanapathy, very good point. (Yeah, strait sed-style regex are a pain once you've gotten use to a powerful regex engine. I normally end up using the sed -r
which enables extended regexs, which are still pretty weak when compared to PCRE.) Splunk is definitely more flexible in that regard 😉
Note that Splunk's SEDCMD
uses PCRE regex, not standard sed
regex.
splunk's SEDCMD works with one event at a time
That's what I was suspecting. I found you can make it work line-by-line by using multi-line regex mode. (I added an answer of my own with an example.)