topic Re: Regex Help in Splunk Search

Regex Help

lufermalgo — Wed, 11 Apr 2018 21:55:39 GMT

Hi community,

Can you please help me create a regular expression that allows me to exclude the leading zeros of a limited string.

I have the following data structure:

05755060010220000002000502234704       20180411135906798L
07095060010220000002300502234705       20180411135906798L
03545070022971000013950504434706       20180411135906798L
00785060060099000011150504034707       20180411135906798L
00745060070142000001550504034708       20180411135906798L
00305070079758000010000502234709       20180411135906798L

This is my regular expression:

^(?<DEP>\d{4})(?<MOV>\d{3})(?<PLU>\d{7})(?<PV>\d{9})(?<EPLU>\d{2})(?<CPLU>\d{2})(?<SEQUEN>\d+)\s*?(?<FEC>\d{16})(?<ORI>.{1})

The first field has a length of 4 digits, but this can contain 1,2,3 or no zero at the beginning, I must clean these zeros they stayed like this:

original = 0575
acceptable= 575

Thank you

Re: Regex Help

p_gurav — Thu, 12 Apr 2018 06:14:19 GMT

Hi,

You can try below regex to exclude preceding zeros:

(?!(0))(?<DEP>\d{3})(?<MOV>\d{3})(?<PLU>\d{7})(?<PV>\d{9})(?<EPLU>\d{2})(?<CPLU>\d{2})(?<SEQUEN>\d+)\s*?(?<FEC>\d{16})(?<ORI>.{1})

Also another way is you can apply one more regex on field DEP to avoid preceding zeros after main search.

Re: Regex Help

FrankVl — Thu, 12 Apr 2018 07:16:38 GMT

To keep it simple and because you don't know how many leading zeros you have, I would keep your regex as you have it, and then strip off the leading zeros in a second step.

e.g.
| rex field=DEP "^0*(?<DEP>[1-9]+)"

or
| eval DEP=ltrim(DEP,"0")

Re: Regex Help

lufermalgo — Thu, 12 Apr 2018 12:29:38 GMT

@FrankVl
thanks for your reply,

My instance is splunkcloud and the support staff does not agree that these transformations are done in search time, it is for this reason that I am looking to make the excllusion of zero initials in each field that I extract.

Re: Regex Help

lufermalgo — Thu, 12 Apr 2018 12:33:36 GMT

@p_gurav

thanks for your reply,

The expression must respect the fixed length and applied to all the fields that are extracted.

Let me explain, the DEP field is of a fine length of 4 digits, this should exclude the amount of zero that appeared but increasing the length of 4 digits.

Ex:

0456 - 456
0089 - 89
0009 - 9

It can not exceed the length because in that case it would be taking digits from another field.

The same for the other fields PLU, MOV, PV, etc ...

Re: Regex Help

FrankVl — Tue, 29 Sep 2020 19:01:24 GMT

Ok, I assumed you were running this in the search bar indeed.

Not too familiar with how things are managed on splunkcloud, but perhaps you can explain how you are configuring this? Is this being done at indextime?

If it is indeed index time using transforms, you could still add an extra transforms stanza that uses the DEP field as the SOURCE_KEY and DEST_KEY and then applies the regex similar to what I suggested.

Re: Regex Help

lufermalgo — Thu, 12 Apr 2018 14:08:11 GMT

@FrankVl
thanks for your reply,

Could you please give me an example?

Re: Regex Help

FrankVl — Fri, 13 Apr 2018 12:30:03 GMT

Again: not entirely sure how this is managed in splunk cloud, but assuming you are applying the regex you shared as an index time extraction, I would add the following to props and transforms.conf to add an extra index time extraction to strip the 0s off the DEP field:

props.conf

[your sourcetype here]
TRANSFORMS-DEPstripzeros = DEP-strip-leading-zeros

transforms.conf

[DEP-strip-leading-zeros]
REGEX = ^0*([1-9]+)
SOURCE_KEY = DEP
DEST_KEY = DEP
FORMAT = $1

Re: Regex Help

lufermalgo — Fri, 13 Apr 2018 12:45:47 GMT

Hi @FrankVl

Thanks for your answer,

When parting something is not going well, when restarting the service it generates me the following error message and the tado keeps conserving the initial zeros.

Undocumented key used in transforms.conf; stanza='DEP-strip-leading-zeros' setting='SOURCE_KEY' key='DEP'
Undocumented key used in transforms.conf; stanza='DEP-strip-leading-zeros' setting='DEST_KEY' key='DEP'
Please resolve these problems by correcting typos in key names, or by adding them to [accepted_keys] in transforms.conf if they are intended.

05755060010220000002000502234704       20180411135906798L
DEP=0575
FEC=20180411135906798
MOV=506
OCPLU=22
OEPLU=50
OPV=000002000
ORI=L
PLU=0010220
SECUEN=34704

Should I consider something else?

Re: Regex Help

FrankVl — Fri, 13 Apr 2018 13:02:01 GMT

My bad, for index time extractions, you cannot refer to already extracted fields, only to a few default KEYS.

What you could of course do, is keep the value incl. leading zeros at index time, but strip off the leading zero's at search time.

So either do that as part of your searches as suggested above, or automatically in props.conf with an eval command (calculated field):
EVAL-DEP=ltrim(DEP,"0")

Re: Regex Help

lufermalgo — Fri, 13 Apr 2018 13:08:10 GMT

Thank you,

I wanted to do it in indexing time to avoid loading when I search, since the data volume is high.

Thanks for supporting me, I'm going to do it in search time to see how it goes.