I am trying to filter events and then apply a sed script to only the events that I want to keep. I want to discard all events that do not contain Keyword1 or Keyword2. I have been able to filter events and use the SEDCMD
, but I have been unsuccessful in using them together. Below is my configuration. How can I make this work? In addition, my regex may be incorrect. I'm not sure how to apply a NOT operator.
props.conf
[source::/path/to/file]
TRANSFORMS-set = setnull
SEDCMD-keep = s/this/that/g
transforms.conf
[setnull]
REGEX = (?!(Keyword1|Keyword2))
DEST_KEY = queue
FORMAT = nullQueue
UPDATE: In the end I used the above solution with regex help provided by @ShaneNewman. As mentioned below, the solution provided by @kristian.kolb works perfectly as well, but I opted for what seems to be a shorter and more efficient method.
UPDATE-2: It appears that SED-*
entries are executed prior to TRANSFORMS-*
. This caused issues with some events not being indexed.
I would probably do it the other way around;
props.conf
[your_source_or_sourcetype]
TRANSFORMS-set = setnull, keepsome
SEDCMD-keep = s/this/that/g
transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[keepsome]
REGEX = (keyword1|keyword2)
DEST_KEY = queue
FORMAT = indexQueue
/K
I would probably do it the other way around;
props.conf
[your_source_or_sourcetype]
TRANSFORMS-set = setnull, keepsome
SEDCMD-keep = s/this/that/g
transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[keepsome]
REGEX = (keyword1|keyword2)
DEST_KEY = queue
FORMAT = indexQueue
/K
This way does work, but it seems inefficient to send everything to the nullQueue first and then pull out what I need. Even though this is the method that is described in the docs. I was hoping to just send only what I don't need to the nullQueue but if there is no difference in performance then maybe it doesn't matter.
REGEX = ^(?!.*(Keyword1|Keyword2)).*$
Please help with this! I believe I'm missing something.
https://answers.splunk.com/answers/334199/index-only-few-fields-and-ignore-the-other-fields.html
I thought I did; I restarted Splunk and it seems to be working as expected. Thanks for the help.
Have you restarted the instance?
Thanks. This seems to work. One thing I noticed is that additional unwanted events were coming through if they contained Keyword1somethingelse so I modified the regex as so ^(?!.*(Keyword1|Keyword2)\b).*$
. However, Keyword1somethingelse is still coming through even though the regex seems to work correctly when I test it in a Splunk search. Any ideas?
Sample events?