topic Re: Filter events and use SEDCMD? in Getting Data In

Filter events and use SEDCMD?

sc0tt — Tue, 29 Oct 2013 14:00:09 GMT

I am trying to filter events and then apply a sed script to only the events that I want to keep. I want to discard all events that do not contain Keyword1 or Keyword2. I have been able to filter events and use the SEDCMD, but I have been unsuccessful in using them together. Below is my configuration. How can I make this work? In addition, my regex may be incorrect. I'm not sure how to apply a NOT operator.

props.conf

[source::/path/to/file]
TRANSFORMS-set = setnull
SEDCMD-keep = s/this/that/g

transforms.conf

[setnull]
REGEX = (?!(Keyword1|Keyword2))
DEST_KEY = queue
FORMAT = nullQueue

UPDATE: In the end I used the above solution with regex help provided by @ShaneNewman. As mentioned below, the solution provided by @kristian.kolb works perfectly as well, but I opted for what seems to be a shorter and more efficient method.

UPDATE-2: It appears that SED-* entries are executed prior to TRANSFORMS-*. This caused issues with some events not being indexed.

Re: Filter events and use SEDCMD?

sowings — Tue, 29 Oct 2013 14:19:20 GMT

Sample events?

Re: Filter events and use SEDCMD?

ShaneNewman — Tue, 29 Oct 2013 15:57:18 GMT

REGEX = ^(?!.*(Keyword1|Keyword2)).*$

Re: Filter events and use SEDCMD?

kristian_kolb — Tue, 29 Oct 2013 16:13:48 GMT

I would probably do it the other way around;

http://docs.splunk.com/Documentation/Splunk/6.0/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest

props.conf

[your_source_or_sourcetype]
TRANSFORMS-set = setnull, keepsome
SEDCMD-keep = s/this/that/g

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keepsome]
REGEX = (keyword1|keyword2)
DEST_KEY = queue
FORMAT = indexQueue

Re: Filter events and use SEDCMD?

sc0tt — Tue, 29 Oct 2013 18:31:54 GMT

This way does work, but it seems inefficient to send everything to the nullQueue first and then pull out what I need. Even though this is the method that is described in the docs. I was hoping to just send only what I don't need to the nullQueue but if there is no difference in performance then maybe it doesn't matter.

Re: Filter events and use SEDCMD?

sc0tt — Tue, 29 Oct 2013 18:48:55 GMT

Thanks. This seems to work. One thing I noticed is that additional unwanted events were coming through if they contained Keyword1somethingelse so I modified the regex as so ^(?!.*(Keyword1|Keyword2)\b).*$. However, Keyword1somethingelse is still coming through even though the regex seems to work correctly when I test it in a Splunk search. Any ideas?

Re: Filter events and use SEDCMD?

ShaneNewman — Tue, 29 Oct 2013 19:14:22 GMT

Have you restarted the instance?

Re: Filter events and use SEDCMD?

sc0tt — Wed, 30 Oct 2013 08:47:53 GMT

I thought I did; I restarted Splunk and it seems to be working as expected. Thanks for the help.

Re: Filter events and use SEDCMD?

allan_newton — Thu, 10 Dec 2015 18:09:15 GMT

Please help with this! I believe I'm missing something.

https://answers.splunk.com/answers/334199/index-only-few-fields-and-ignore-the-other-fields.html