Hi community,
Can you please help me create a regular expression that allows me to exclude the leading zeros of a limited string.
I have the following data structure:
05755060010220000002000502234704 20180411135906798L
07095060010220000002300502234705 20180411135906798L
03545070022971000013950504434706 20180411135906798L
00785060060099000011150504034707 20180411135906798L
00745060070142000001550504034708 20180411135906798L
00305070079758000010000502234709 20180411135906798L
This is my regular expression:
^(?<DEP>\d{4})(?<MOV>\d{3})(?<PLU>\d{7})(?<PV>\d{9})(?<EPLU>\d{2})(?<CPLU>\d{2})(?<SEQUEN>\d+)\s*?(?<FEC>\d{16})(?<ORI>.{1})
The first field has a length of 4 digits, but this can contain 1,2,3 or no zero at the beginning, I must clean these zeros they stayed like this:
original = 0575
acceptable= 575
Thank you
To keep it simple and because you don't know how many leading zeros you have, I would keep your regex as you have it, and then strip off the leading zeros in a second step.
e.g.
| rex field=DEP "^0*(?<DEP>[1-9]+)"
or
| eval DEP=ltrim(DEP,"0")
@FrankVl
thanks for your reply,
My instance is splunkcloud and the support staff does not agree that these transformations are done in search time, it is for this reason that I am looking to make the excllusion of zero initials in each field that I extract.
Ok, I assumed you were running this in the search bar indeed.
Not too familiar with how things are managed on splunkcloud, but perhaps you can explain how you are configuring this? Is this being done at indextime?
If it is indeed index time using transforms, you could still add an extra transforms stanza that uses the DEP field as the SOURCE_KEY and DEST_KEY and then applies the regex similar to what I suggested.
@FrankVl
thanks for your reply,
Could you please give me an example?
Again: not entirely sure how this is managed in splunk cloud, but assuming you are applying the regex you shared as an index time extraction, I would add the following to props and transforms.conf to add an extra index time extraction to strip the 0s off the DEP field:
props.conf
[your sourcetype here]
TRANSFORMS-DEPstripzeros = DEP-strip-leading-zeros
transforms.conf
[DEP-strip-leading-zeros]
REGEX = ^0*([1-9]+)
SOURCE_KEY = DEP
DEST_KEY = DEP
FORMAT = $1
Hi @FrankVl
Thanks for your answer,
When parting something is not going well, when restarting the service it generates me the following error message and the tado keeps conserving the initial zeros.
Undocumented key used in transforms.conf; stanza='DEP-strip-leading-zeros' setting='SOURCE_KEY' key='DEP'
Undocumented key used in transforms.conf; stanza='DEP-strip-leading-zeros' setting='DEST_KEY' key='DEP'
Please resolve these problems by correcting typos in key names, or by adding them to [accepted_keys] in transforms.conf if they are intended.
05755060010220000002000502234704 20180411135906798L
DEP=0575
FEC=20180411135906798
MOV=506
OCPLU=22
OEPLU=50
OPV=000002000
ORI=L
PLU=0010220
SECUEN=34704
Should I consider something else?
My bad, for index time extractions, you cannot refer to already extracted fields, only to a few default KEYS.
What you could of course do, is keep the value incl. leading zeros at index time, but strip off the leading zero's at search time.
So either do that as part of your searches as suggested above, or automatically in props.conf with an eval command (calculated field):
EVAL-DEP=ltrim(DEP,"0")
Thank you,
I wanted to do it in indexing time to avoid loading when I search, since the data volume is high.
Thanks for supporting me, I'm going to do it in search time to see how it goes.
Hi,
You can try below regex to exclude preceding zeros:
(?!(0))(?<DEP>\d{3})(?<MOV>\d{3})(?<PLU>\d{7})(?<PV>\d{9})(?<EPLU>\d{2})(?<CPLU>\d{2})(?<SEQUEN>\d+)\s*?(?<FEC>\d{16})(?<ORI>.{1})
Also another way is you can apply one more regex on field DEP
to avoid preceding zeros after main search.
@p_gurav
thanks for your reply,
The expression must respect the fixed length and applied to all the fields that are extracted.
Let me explain, the DEP field is of a fine length of 4 digits, this should exclude the amount of zero that appeared but increasing the length of 4 digits.
Ex:
0456 - 456
0089 - 89
0009 - 9
It can not exceed the length because in that case it would be taking digits from another field.
The same for the other fields PLU, MOV, PV, etc ...