Getting Data In

Filter events and use SEDCMD?

sc0tt
Builder

I am trying to filter events and then apply a sed script to only the events that I want to keep. I want to discard all events that do not contain Keyword1 or Keyword2. I have been able to filter events and use the SEDCMD, but I have been unsuccessful in using them together. Below is my configuration. How can I make this work? In addition, my regex may be incorrect. I'm not sure how to apply a NOT operator.

props.conf

[source::/path/to/file]
TRANSFORMS-set = setnull
SEDCMD-keep = s/this/that/g

transforms.conf

[setnull]
REGEX = (?!(Keyword1|Keyword2))
DEST_KEY = queue
FORMAT = nullQueue

UPDATE: In the end I used the above solution with regex help provided by @ShaneNewman. As mentioned below, the solution provided by @kristian.kolb works perfectly as well, but I opted for what seems to be a shorter and more efficient method.

UPDATE-2: It appears that SED-* entries are executed prior to TRANSFORMS-*. This caused issues with some events not being indexed.

Tags (2)
1 Solution

kristian_kolb
Ultra Champion

I would probably do it the other way around;

http://docs.splunk.com/Documentation/Splunk/6.0/Forwarding/Routeandfilterdatad#Keep_specific_events_...

props.conf

[your_source_or_sourcetype]
TRANSFORMS-set = setnull, keepsome
SEDCMD-keep = s/this/that/g

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keepsome]
REGEX = (keyword1|keyword2)
DEST_KEY = queue
FORMAT = indexQueue

/K

View solution in original post

kristian_kolb
Ultra Champion

I would probably do it the other way around;

http://docs.splunk.com/Documentation/Splunk/6.0/Forwarding/Routeandfilterdatad#Keep_specific_events_...

props.conf

[your_source_or_sourcetype]
TRANSFORMS-set = setnull, keepsome
SEDCMD-keep = s/this/that/g

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keepsome]
REGEX = (keyword1|keyword2)
DEST_KEY = queue
FORMAT = indexQueue

/K

sc0tt
Builder

This way does work, but it seems inefficient to send everything to the nullQueue first and then pull out what I need. Even though this is the method that is described in the docs. I was hoping to just send only what I don't need to the nullQueue but if there is no difference in performance then maybe it doesn't matter.

0 Karma

ShaneNewman
Motivator
REGEX = ^(?!.*(Keyword1|Keyword2)).*$

allan_newton
Path Finder
0 Karma

sc0tt
Builder

I thought I did; I restarted Splunk and it seems to be working as expected. Thanks for the help.

0 Karma

ShaneNewman
Motivator

Have you restarted the instance?

0 Karma

sc0tt
Builder

Thanks. This seems to work. One thing I noticed is that additional unwanted events were coming through if they contained Keyword1somethingelse so I modified the regex as so ^(?!.*(Keyword1|Keyword2)\b).*$. However, Keyword1somethingelse is still coming through even though the regex seems to work correctly when I test it in a Splunk search. Any ideas?

0 Karma

sowings
Splunk Employee
Splunk Employee

Sample events?

0 Karma
Get Updates on the Splunk Community!

Data Management Digest – December 2025

Welcome to the December edition of Data Management Digest! As we continue our journey of data innovation, the ...

Index This | What is broken 80% of the time by February?

December 2025 Edition   Hayyy Splunk Education Enthusiasts and the Eternally Curious!    We’re back with this ...

Unlock Faster Time-to-Value on Edge and Ingest Processor with New SPL2 Pipeline ...

Hello Splunk Community,   We're thrilled to share an exciting update that will help you manage your data more ...