topic Re: eliminate unnecessary values when indexing in Getting Data In

eliminate unnecessary values when indexing

efaundez — Thu, 28 Jun 2018 16:50:03 GMT

good morning

I want to ignore certain elements of a log when indexing them, for example:

field0 | x | x | x | x | x | field6 | field7 | field8 | x | x | x | field12 | field13 | field14 | field15 | field16 | field17 | x | field19 | field20 | x | x | x | x | x | x | field27 | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | field48

I have many values in this line of events and I just want the FIELDXX values to be indexed, and the values between | x | do not. I know that a whole line of events can be ignored using the transform.conf, but in this case I only want certain values. Is this possible?

regards

Re: eliminate unnecessary values when indexing

richgalloway — Thu, 28 Jun 2018 16:54:03 GMT

Depending on what "x" really is, you may be able to use SEDCMD to edit them out.

Re: eliminate unnecessary values when indexing

efaundez — Thu, 28 Jun 2018 16:59:32 GMT

thanks for the answer, is there any documentation or example to validate and test?

regards

Re: eliminate unnecessary values when indexing

efaundez — Thu, 28 Jun 2018 17:00:26 GMT

most of the field1, field2, field3 ... are numeric and some dates

Re: eliminate unnecessary values when indexing

lacastillo — Thu, 28 Jun 2018 17:42:20 GMT

Here's the documentation for SEDCMD in props.conf

https://docs.splunk.com/Documentation/Splunk/7.1.1/Data/Anonymizedata#Define_the_sed_script_in_props.conf

Re: eliminate unnecessary values when indexing

efaundez — Thu, 28 Jun 2018 17:48:08 GMT

thanks for the reply

I will do the relevant tests.

Re: eliminate unnecessary values when indexing

efaundez — Thu, 28 Jun 2018 18:02:58 GMT

it is not required to mask the data, it is necessary to omit and not replace it with another value or text.

Re: eliminate unnecessary values when indexing

richgalloway — Fri, 29 Jun 2018 12:04:25 GMT

Replace it with an empty string.

Re: eliminate unnecessary values when indexing

woodcock — Sat, 30 Jun 2018 15:23:58 GMT

If you must do this in Splunk (on your Indexers), you can do it with SEDCMD. Here is a proof of concept:

| makeresults 
| eval _raw="field0 | x | x | x | x | x | field6 | field7 | field8 | x | x | x | field12 | field13 | field14 | field15 | field16 | field17 | x | field19 | field20 | x | x | x | x | x | x | field27 | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | field48"
| eval raw2=_raw
| rex field=raw2 mode=sed "s/\s*x\s*(?=|)//g"
| rex field=_raw mode=sed "s/^([^|]*(?=|))\|(?:[^|]*(?=|)\|){5}((?:[^|]*(?=|)\|){3})(?:[^|]*(?=|)\|){3}((?:[^|]*(?=|)\|){6})(?:[^|]*(?=|)\|){1}((?:[^|]*(?=|)\|){2})(?:[^|]*(?=|)\|){6}((?:[^|]*(?=|)\|){1})(?:[^|]*(?=|)\|){20}(.*)$/\1||||||\2|||\3|\4||||||\5||||||||||||||||||||\6/"
| eval TEST=if((raw2==_raw), "GOOD!", "ERROR!")

So your props.conf line would look like this:

SEDCMD-strip_some_PSV_values = s/^([^|]*(?=|))\|(?:[^|]*(?=|)\|){5}((?:[^|]*(?=|)\|){3})(?:[^|]*(?=|)\|){3}((?:[^|]*(?=|)\|){6})(?:[^|]*(?=|)\|){1}((?:[^|]*(?=|)\|){2})(?:[^|]*(?=|)\|){6}((?:[^|]*(?=|)\|){1})(?:[^|]*(?=|)\|){20}(.*)$/\1||||||\2|||\3|\4||||||\5||||||||||||||||||||\6/

Re: eliminate unnecessary values when indexing

efaundez — Tue, 03 Jul 2018 13:15:37 GMT

good morning

Thanks for your answer, I'll do the relevant tests