Getting Data In

Does SEDCMD work line-by-line or on the entire event?

Super Champion

Out of the box, the unix sed command operates on a line-by-line basis. Is this the same for the SEDCMD setting in props.conf? Or does it operate on the entire event?


Just for the sake of an example. Say I'm using the following unix command to filter out comment lines (lines starting with a ";"). (Technically, this example only clears the comment lines rather than remove them, "grep" would be more obvious choice of unix commands, but splunk doesn't have a GREPCMD, so it's a mute point.)

cat myfile |  sed -re 's/^\s*;.*$//'

Will that work the same way as the following props.conf entry:

SEDCMD-drop_comments = s/^\*;.*$//
Tags (1)
1 Solution

Splunk Employee
Splunk Employee

splunk's SEDCMD works with one event at a time

View solution in original post

Super Champion

To get the SEDCMD to work on a line-by-line basis, you can use the following:

SEDCMD-drop_comments = s/(?m)^\*;.*$//g

The difference is:

  1. The (?m) enables multi-line mode in regex engine. This causes ^ and $ to now mean start-of-line and end-of-line, rather than start-of-event and end-of-event.
  2. The g on the end tells the sed command to be applied as many times as possible; therefore the replacement will occur more than once. So without the g, only the first matching line would be replaced instead of all the lines.

Super Champion

@gkanapathy, very good point. (Yeah, strait sed-style regex are a pain once you've gotten use to a powerful regex engine. I normally end up using the sed -r which enables extended regexs, which are still pretty weak when compared to PCRE.) Splunk is definitely more flexible in that regard 😉

0 Karma

Splunk Employee
Splunk Employee

Note that Splunk's SEDCMD uses PCRE regex, not standard sed regex.

0 Karma

Splunk Employee
Splunk Employee

splunk's SEDCMD works with one event at a time

View solution in original post

Super Champion

That's what I was suspecting. I found you can make it work line-by-line by using multi-line regex mode. (I added an answer of my own with an example.)

0 Karma