Getting Data In

eliminate unnecessary values when indexing

efaundez
Path Finder

good morning

     I want to ignore certain elements of a log when indexing them, for example:

field0 | x | x | x | x | x | field6 | field7 | field8 | x | x | x | field12 | field13 | field14 | field15 | field16 | field17 | x | field19 | field20 | x | x | x | x | x | x | field27 | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | field48

    I have many values in this line of events and I just want the FIELDXX values to be indexed, and the values between | x | do not. I know that a whole line of events can be ignored using the transform.conf, but in this case I only want certain values. Is this possible?

regards

0 Karma

woodcock
Esteemed Legend

If you must do this in Splunk (on your Indexers), you can do it with SEDCMD. Here is a proof of concept:

| makeresults 
| eval _raw="field0 | x | x | x | x | x | field6 | field7 | field8 | x | x | x | field12 | field13 | field14 | field15 | field16 | field17 | x | field19 | field20 | x | x | x | x | x | x | field27 | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | field48"
| eval raw2=_raw
| rex field=raw2 mode=sed "s/\s*x\s*(?=|)//g"
| rex field=_raw mode=sed "s/^([^|]*(?=|))\|(?:[^|]*(?=|)\|){5}((?:[^|]*(?=|)\|){3})(?:[^|]*(?=|)\|){3}((?:[^|]*(?=|)\|){6})(?:[^|]*(?=|)\|){1}((?:[^|]*(?=|)\|){2})(?:[^|]*(?=|)\|){6}((?:[^|]*(?=|)\|){1})(?:[^|]*(?=|)\|){20}(.*)$/\1||||||\2|||\3|\4||||||\5||||||||||||||||||||\6/"
| eval TEST=if((raw2==_raw), "GOOD!", "ERROR!")

So your props.conf line would look like this:

SEDCMD-strip_some_PSV_values = s/^([^|]*(?=|))\|(?:[^|]*(?=|)\|){5}((?:[^|]*(?=|)\|){3})(?:[^|]*(?=|)\|){3}((?:[^|]*(?=|)\|){6})(?:[^|]*(?=|)\|){1}((?:[^|]*(?=|)\|){2})(?:[^|]*(?=|)\|){6}((?:[^|]*(?=|)\|){1})(?:[^|]*(?=|)\|){20}(.*)$/\1||||||\2|||\3|\4||||||\5||||||||||||||||||||\6/
0 Karma

efaundez
Path Finder

good morning

   Thanks for your answer, I'll do the relevant tests

0 Karma

lacastillo
Path Finder

efaundez
Path Finder

it is not required to mask the data, it is necessary to omit and not replace it with another value or text.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Replace it with an empty string.

---
If this reply helps you, Karma would be appreciated.

efaundez
Path Finder

thanks for the reply

    I will do the relevant tests.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Depending on what "x" really is, you may be able to use SEDCMD to edit them out.

---
If this reply helps you, Karma would be appreciated.
0 Karma

efaundez
Path Finder

thanks for the answer, is there any documentation or example to validate and test?

regards

0 Karma

efaundez
Path Finder

most of the field1, field2, field3 ... are numeric and some dates

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...