Getting Data In

Filter events and use SEDCMD?

sc0tt
Builder

I am trying to filter events and then apply a sed script to only the events that I want to keep. I want to discard all events that do not contain Keyword1 or Keyword2. I have been able to filter events and use the SEDCMD, but I have been unsuccessful in using them together. Below is my configuration. How can I make this work? In addition, my regex may be incorrect. I'm not sure how to apply a NOT operator.

props.conf

[source::/path/to/file]
TRANSFORMS-set = setnull
SEDCMD-keep = s/this/that/g

transforms.conf

[setnull]
REGEX = (?!(Keyword1|Keyword2))
DEST_KEY = queue
FORMAT = nullQueue

UPDATE: In the end I used the above solution with regex help provided by @ShaneNewman. As mentioned below, the solution provided by @kristian.kolb works perfectly as well, but I opted for what seems to be a shorter and more efficient method.

UPDATE-2: It appears that SED-* entries are executed prior to TRANSFORMS-*. This caused issues with some events not being indexed.

Tags (2)
1 Solution

kristian_kolb
Ultra Champion

I would probably do it the other way around;

http://docs.splunk.com/Documentation/Splunk/6.0/Forwarding/Routeandfilterdatad#Keep_specific_events_...

props.conf

[your_source_or_sourcetype]
TRANSFORMS-set = setnull, keepsome
SEDCMD-keep = s/this/that/g

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keepsome]
REGEX = (keyword1|keyword2)
DEST_KEY = queue
FORMAT = indexQueue

/K

View solution in original post

kristian_kolb
Ultra Champion

I would probably do it the other way around;

http://docs.splunk.com/Documentation/Splunk/6.0/Forwarding/Routeandfilterdatad#Keep_specific_events_...

props.conf

[your_source_or_sourcetype]
TRANSFORMS-set = setnull, keepsome
SEDCMD-keep = s/this/that/g

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keepsome]
REGEX = (keyword1|keyword2)
DEST_KEY = queue
FORMAT = indexQueue

/K

sc0tt
Builder

This way does work, but it seems inefficient to send everything to the nullQueue first and then pull out what I need. Even though this is the method that is described in the docs. I was hoping to just send only what I don't need to the nullQueue but if there is no difference in performance then maybe it doesn't matter.

0 Karma

ShaneNewman
Motivator
REGEX = ^(?!.*(Keyword1|Keyword2)).*$

allan_newton
Path Finder
0 Karma

sc0tt
Builder

I thought I did; I restarted Splunk and it seems to be working as expected. Thanks for the help.

0 Karma

ShaneNewman
Motivator

Have you restarted the instance?

0 Karma

sc0tt
Builder

Thanks. This seems to work. One thing I noticed is that additional unwanted events were coming through if they contained Keyword1somethingelse so I modified the regex as so ^(?!.*(Keyword1|Keyword2)\b).*$. However, Keyword1somethingelse is still coming through even though the regex seems to work correctly when I test it in a Splunk search. Any ideas?

0 Karma

sowings
Splunk Employee
Splunk Employee

Sample events?

0 Karma
Get Updates on the Splunk Community!

Dashboard Studio Challenge - Learn New Tricks, Showcase Your Skills, and Win Prizes!

Reimagine what you can do with your dashboards. Dashboard Studio is Splunk’s newest dashboard builder to ...

Introducing Edge Processor: Next Gen Data Transformation

We get it - not only can it take a lot of time, money and resources to get data into Splunk, but it also takes ...

Take the 2021 Splunk Career Survey for $50 in Amazon Cash

Help us learn about how Splunk has impacted your career by taking the 2021 Splunk Career Survey. Last year’s ...