Splunk Search

Post search filtering

Marinus
Communicator

Is there a way to apply a SED like filter after a search. The plumbing is there to filter and sanitize data going into the indexer. You could achieve this with a custom search command but is there another way? Would be handy if there was a props config.

Tags (1)
1 Solution

Lowell
Super Champion

You have two options. Either use the | rex mode=sed "s/find/replace/g" at runtime, or use a SEDCMD at index time, like so:

In props.conf:

[my_source_type]
SEDCMD = s/find/replace/g

Now, if if you are asking if you case setup something like a SEDCMD that gets run at search time, then the answer is no.

Here is a possible workaround: If your sed command is really long and ugly and you simply don't want to see it or don't want to repeat it in multiple searches, then I would suggest that you create a macro with your rex. I think that's the best you can do.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

There are some tricks, but I would just stick with expressing them in the search query language (with either rex or eval for your case), using macros if that makes things clearer or easier to manage.

Note that the existing "automatic" extractions and lookups at search time are not merely syntactic replacements for rex and lookup commands. They also allow you to perform efficient searches on the new fields by reverse-mapping the requested fields to original indexed values. This would not be possible with a general sed command.

But you can make things automatic. Note that doing this will also probably make it incredibly confusing to search for items, even if you're looking right at them:

props.conf:

[mysourcetype]
REPORT-raw = changeraw
FIELDALIAS-raw = _mychangedraw AS _raw

transforms.conf:

[changeraw]
SOURCE_KEY = _raw
REGEX = ^(?<_mychangedraw>.{1,64})
CLEAN_KEYS = false

This only works however if the change you're trying to make to a field is simply slicing out one part of it. If you need to do something more complicated, you do need a custom script, but you can use a lookup to get it to run automatically. A lookup, like an extraction, won't overwrite an initially returned field, so you'd do similar to the above:

[mysourcetype]
LOOKUP-raw = mycustomlookupscript _raw OUTPUT _mychangedraw
FIELDALIAS-raw = _mychangedraw AS _raw

[mycustomlookupscript]
external_cmd = mylookupscript.py
fields_list = _raw,_mychangedraw

Well, you have to have an external script instead of using Splunk to set regexes here.

0 Karma

Lowell
Super Champion

You have two options. Either use the | rex mode=sed "s/find/replace/g" at runtime, or use a SEDCMD at index time, like so:

In props.conf:

[my_source_type]
SEDCMD = s/find/replace/g

Now, if if you are asking if you case setup something like a SEDCMD that gets run at search time, then the answer is no.

Here is a possible workaround: If your sed command is really long and ugly and you simply don't want to see it or don't want to repeat it in multiple searches, then I would suggest that you create a macro with your rex. I think that's the best you can do.

Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...