Getting Data In

eliminate unnecessary values when indexing

efaundez
Path Finder

good morning

     I want to ignore certain elements of a log when indexing them, for example:

field0 | x | x | x | x | x | field6 | field7 | field8 | x | x | x | field12 | field13 | field14 | field15 | field16 | field17 | x | field19 | field20 | x | x | x | x | x | x | field27 | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | field48

    I have many values in this line of events and I just want the FIELDXX values to be indexed, and the values between | x | do not. I know that a whole line of events can be ignored using the transform.conf, but in this case I only want certain values. Is this possible?

regards

0 Karma

woodcock
Esteemed Legend

If you must do this in Splunk (on your Indexers), you can do it with SEDCMD. Here is a proof of concept:

| makeresults 
| eval _raw="field0 | x | x | x | x | x | field6 | field7 | field8 | x | x | x | field12 | field13 | field14 | field15 | field16 | field17 | x | field19 | field20 | x | x | x | x | x | x | field27 | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | field48"
| eval raw2=_raw
| rex field=raw2 mode=sed "s/\s*x\s*(?=|)//g"
| rex field=_raw mode=sed "s/^([^|]*(?=|))\|(?:[^|]*(?=|)\|){5}((?:[^|]*(?=|)\|){3})(?:[^|]*(?=|)\|){3}((?:[^|]*(?=|)\|){6})(?:[^|]*(?=|)\|){1}((?:[^|]*(?=|)\|){2})(?:[^|]*(?=|)\|){6}((?:[^|]*(?=|)\|){1})(?:[^|]*(?=|)\|){20}(.*)$/\1||||||\2|||\3|\4||||||\5||||||||||||||||||||\6/"
| eval TEST=if((raw2==_raw), "GOOD!", "ERROR!")

So your props.conf line would look like this:

SEDCMD-strip_some_PSV_values = s/^([^|]*(?=|))\|(?:[^|]*(?=|)\|){5}((?:[^|]*(?=|)\|){3})(?:[^|]*(?=|)\|){3}((?:[^|]*(?=|)\|){6})(?:[^|]*(?=|)\|){1}((?:[^|]*(?=|)\|){2})(?:[^|]*(?=|)\|){6}((?:[^|]*(?=|)\|){1})(?:[^|]*(?=|)\|){20}(.*)$/\1||||||\2|||\3|\4||||||\5||||||||||||||||||||\6/
0 Karma

efaundez
Path Finder

good morning

   Thanks for your answer, I'll do the relevant tests

0 Karma

lacastillo
Path Finder

efaundez
Path Finder

it is not required to mask the data, it is necessary to omit and not replace it with another value or text.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Replace it with an empty string.

---
If this reply helps you, Karma would be appreciated.

efaundez
Path Finder

thanks for the reply

    I will do the relevant tests.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Depending on what "x" really is, you may be able to use SEDCMD to edit them out.

---
If this reply helps you, Karma would be appreciated.
0 Karma

efaundez
Path Finder

thanks for the answer, is there any documentation or example to validate and test?

regards

0 Karma

efaundez
Path Finder

most of the field1, field2, field3 ... are numeric and some dates

0 Karma
Get Updates on the Splunk Community!

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...