eliminate unnecessary values when indexing

[efaundez]

good morning

     I want to ignore certain elements of a log when indexing them, for example:

field0 | x | x | x | x | x | field6 | field7 | field8 | x | x | x | field12 | field13 | field14 | field15 | field16 | field17 | x | field19 | field20 | x | x | x | x | x | x | field27 | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | field48

    I have many values in this line of events and I just want the FIELDXX values to be indexed, and the values between | x | do not. I know that a whole line of events can be ignored using the transform.conf, but in this case I only want certain values. Is this possible?

regards

[woodcock]

If you must do this in Splunk (on your Indexers), you can do it with SEDCMD. Here is a proof of concept:

| makeresults 
| eval _raw="field0 | x | x | x | x | x | field6 | field7 | field8 | x | x | x | field12 | field13 | field14 | field15 | field16 | field17 | x | field19 | field20 | x | x | x | x | x | x | field27 | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | field48"
| eval raw2=_raw
| rex field=raw2 mode=sed "s/\s*x\s*(?=|)//g"
| rex field=_raw mode=sed "s/^([^|]*(?=|))\|(?:[^|]*(?=|)\|){5}((?:[^|]*(?=|)\|){3})(?:[^|]*(?=|)\|){3}((?:[^|]*(?=|)\|){6})(?:[^|]*(?=|)\|){1}((?:[^|]*(?=|)\|){2})(?:[^|]*(?=|)\|){6}((?:[^|]*(?=|)\|){1})(?:[^|]*(?=|)\|){20}(.*)$/\1||||||\2|||\3|\4||||||\5||||||||||||||||||||\6/"
| eval TEST=if((raw2==_raw), "GOOD!", "ERROR!")

So your props.conf line would look like this:

SEDCMD-strip_some_PSV_values = s/^([^|]*(?=|))\|(?:[^|]*(?=|)\|){5}((?:[^|]*(?=|)\|){3})(?:[^|]*(?=|)\|){3}((?:[^|]*(?=|)\|){6})(?:[^|]*(?=|)\|){1}((?:[^|]*(?=|)\|){2})(?:[^|]*(?=|)\|){6}((?:[^|]*(?=|)\|){1})(?:[^|]*(?=|)\|){20}(.*)$/\1||||||\2|||\3|\4||||||\5||||||||||||||||||||\6/

[efaundez]

good morning

   Thanks for your answer, I'll do the relevant tests

[lacastillo]

Here's the documentation for SEDCMD in props.conf

https://docs.splunk.com/Documentation/Splunk/7.1.1/Data/Anonymizedata#Define_the_sed_script_in_props...

[efaundez]

it is not required to mask the data, it is necessary to omit and not replace it with another value or text.

[richgalloway]

Replace it with an empty string.

[efaundez]

thanks for the reply

    I will do the relevant tests.

[richgalloway]

Depending on what "x" really is, you may be able to use SEDCMD to edit them out.

[efaundez]

thanks for the answer, is there any documentation or example to validate and test?

regards

[efaundez]

most of the field1, field2, field3 ... are numeric and some dates