Getting Data In

eliminate unnecessary values when indexing

Path Finder

good morning

     I want to ignore certain elements of a log when indexing them, for example:

field0 | x | x | x | x | x | field6 | field7 | field8 | x | x | x | field12 | field13 | field14 | field15 | field16 | field17 | x | field19 | field20 | x | x | x | x | x | x | field27 | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | field48

    I have many values in this line of events and I just want the FIELDXX values to be indexed, and the values between | x | do not. I know that a whole line of events can be ignored using the transform.conf, but in this case I only want certain values. Is this possible?

regards

0 Karma

Esteemed Legend

If you must do this in Splunk (on your Indexers), you can do it with SEDCMD. Here is a proof of concept:

| makeresults 
| eval _raw="field0 | x | x | x | x | x | field6 | field7 | field8 | x | x | x | field12 | field13 | field14 | field15 | field16 | field17 | x | field19 | field20 | x | x | x | x | x | x | field27 | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | field48"
| eval raw2=_raw
| rex field=raw2 mode=sed "s/\s*x\s*(?=|)//g"
| rex field=_raw mode=sed "s/^([^|]*(?=|))\|(?:[^|]*(?=|)\|){5}((?:[^|]*(?=|)\|){3})(?:[^|]*(?=|)\|){3}((?:[^|]*(?=|)\|){6})(?:[^|]*(?=|)\|){1}((?:[^|]*(?=|)\|){2})(?:[^|]*(?=|)\|){6}((?:[^|]*(?=|)\|){1})(?:[^|]*(?=|)\|){20}(.*)$/\1||||||\2|||\3|\4||||||\5||||||||||||||||||||\6/"
| eval TEST=if((raw2==_raw), "GOOD!", "ERROR!")

So your props.conf line would look like this:

SEDCMD-strip_some_PSV_values = s/^([^|]*(?=|))\|(?:[^|]*(?=|)\|){5}((?:[^|]*(?=|)\|){3})(?:[^|]*(?=|)\|){3}((?:[^|]*(?=|)\|){6})(?:[^|]*(?=|)\|){1}((?:[^|]*(?=|)\|){2})(?:[^|]*(?=|)\|){6}((?:[^|]*(?=|)\|){1})(?:[^|]*(?=|)\|){20}(.*)$/\1||||||\2|||\3|\4||||||\5||||||||||||||||||||\6/
0 Karma

Path Finder

good morning

   Thanks for your answer, I'll do the relevant tests

0 Karma

Path Finder

Path Finder

it is not required to mask the data, it is necessary to omit and not replace it with another value or text.

0 Karma

SplunkTrust
SplunkTrust

Replace it with an empty string.

---
If this reply helps you, an upvote would be appreciated.

Path Finder

thanks for the reply

    I will do the relevant tests.

0 Karma

SplunkTrust
SplunkTrust

Depending on what "x" really is, you may be able to use SEDCMD to edit them out.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

Path Finder

thanks for the answer, is there any documentation or example to validate and test?

regards

0 Karma

Path Finder

most of the field1, field2, field3 ... are numeric and some dates

0 Karma