topic event contains same field with different values in Splunk Search

event contains same field with different values

Mike6960 — Tue, 29 Sep 2020 19:45:40 GMT

My events contain teh same fieldnames multiple times with different values.
I.E.
< active_recip="9" deliv_recip="0" hard_bounced="4" hostname="clnpniv.local *" soft_bounced="0"/>
< active_recip="3" deliv_recip="0" hard_bounced="1" hostname="clnpdmz.local *" soft_bounced="0"/>

I tried a field extraction but then only one value is recognized as a field. How can I solve this? I searched online voor MV etc. but I am a bit stuck

Re: event contains same field with different values

diogofgm — Mon, 04 Jun 2018 11:01:07 GMT

You can do the extraction using props/transforms and add the MV_ADD = true in the transforms.conf

transforms.conf

[my_extraction]
REGEX = <regular expression>
MV_ADD = true

props.conf

[my_sourcetype]

REPORT-my_extraction = my_extraction

Re: event contains same field with different values

Sukisen1981 — Tue, 29 Sep 2020 19:45:42 GMT

Hi,

Your issue should be solved using this -
max_match
Syntax: max_match=
Description: Controls the number of times the regex is matched. If greater than 1, the resulting fields are multivalued fields.
Default: 1, use 0 to mean unlimited.
Ref. https://docs.splunk.com/Documentation/Splunk/7.1.1/SearchReference/Rex

just append max_match=0 to your rex and it should fetch all values for the same regex field

Re: event contains same field with different values

Mike6960 — Mon, 04 Jun 2018 11:26:21 GMT

append max_match=0 ?. Where? In the fieldextraction editor?

Re: event contains same field with different values

Sukisen1981 — Tue, 29 Sep 2020 19:45:45 GMT

refer to the documentation above -
something like
sourcetype=linux_secure port "failed password" | rex "\s+(?port \d+)" max_match=0 | stats
so in this case your extracted field , ports, if there are more than one value for ports ALL of them will be fetched under ports.
You can re-use the field extractor regex if you don;t want to write your own regex , and similarly replace the above rex with the rex generated by the field extractor appended by max_match=0

Re: event contains same field with different values

Mike6960 — Mon, 04 Jun 2018 12:02:47 GMT

Ok, I understand what you, but when I use the regex from the editor then I get the message 'unbalanced quotes'
Here is the rex:
^<\?\w+\s+\w+="\d+.\d+"\s+\w+="\w+-\d+"\s+\w+="\w+"\?>\s+<\w+\s+\w+\w+="\w+\w+"\s+\w+="\w+\s+\d+.\d+.\d+-\d+"\s+\w+="\w+.\w+.\w+"\s+\w+="\d+">\s+<\w+\w+\s+\w+\w+="\d+"\s+\w+\w+="\d+"\s+\w+\w+="\d+"\s+\w+_\w+="\d+"\s+\w+="(?P[^*]+)

Re: event contains same field with different values

Sukisen1981 — Tue, 29 Sep 2020 19:45:47 GMT

do we only need to extract active_recip to soft_bounced, ie, the 4-5 fileds as shown in your sample events?

Re: event contains same field with different values

Mike6960 — Mon, 04 Jun 2018 15:21:44 GMT

No, I need to extract the values for hard_bounced en for hostname

Re: event contains same field with different values

Sukisen1981 — Mon, 04 Jun 2018 15:41:51 GMT

Try this - <your index>|rex field=_raw "hard_bounced=\"+(?<hard>.*?)\"+" max_match=0 | rex field=_raw "hostname=\"+(?<hostname>.*?)\"+" max_match=0 | table hard,hostname

Re: event contains same field with different values

Sukisen1981 — Mon, 04 Jun 2018 16:06:17 GMT

On second thoughts you might want them in separate row , try this if that is the case| rex field=_raw "hard_bounced=\"+(?.*?)\"+" max_match=0 | rex field=_raw "hostname=\"+(?.*?)\"+" max_match=0 | eval fields = mvzip(hard,hostname) | mvexpand fields | rex field=fields "(?\w+),(?\w+)" | table _time hard hostname
Try both this one and the above..one of this is what you need

Re: event contains same field with different values

Mike6960 — Tue, 12 Jun 2018 08:59:32 GMT

Hai @Sukisen1981 , if I use the second one I get the message 'Error in 'rex' command: Encountered the following error while compiling the regex 'hard_bounced="+(?.?)"+': Regex: unrecognized character after (? or (?- '

Re: event contains same field with different values

Sukisen1981 — Tue, 12 Jun 2018 16:10:15 GMT

Hi, the code got corrupted while copying, essentially the second query is the same as the first one till mvzip starts...use this

<your index>|rex field=_raw "hard_bounced=\"+(?<hard>.*?)\"+" max_match=0 | rex field=_raw "hostname=\"+(?<hostname>.*?)\"+" max_match=0 | eval fields = mvzip(hard,hostname) 
| mvexpand fields 
| rex field=fields "(?\w+),(?\w+)" 
| table _time hard hostname

Just try to fit the entire code in a continuous line, if you encounter errors., this will work

Re: event contains same field with different values

Mike6960 — Wed, 13 Jun 2018 08:47:45 GMT

sadly, still the same error. Don't see a difference between the codes by the way

Re: event contains same field with different values

Sukisen1981 — Fri, 15 Jun 2018 09:45:43 GMT

Hi - Sorry , I once again copied the same code....i am pasting the correct code now. The difference is in the rex field statement, before \w+ the individual field names have to be mentioned, it is just a minor rex syntax issue.
I apologise once again for pasting the same wrong code twice|rex field=_raw "hard_bounced=\"+(?.*?)\"+" max_match=0 | rex field=_raw "hostname=\"+(?.*?)\"+" max_match=0 | eval fields = mvzip(hard,hostname) | mvexpand fields | rex field=fields "(?\w+),(?\w+)" | table _time hard hostname

Re: event contains same field with different values

Sukisen1981 — Tue, 29 Sep 2020 20:00:46 GMT

i see it again the code is not getting pasted properly again.....look at the below

Re: event contains same field with different values

Sukisen1981 — Fri, 15 Jun 2018 09:48:33 GMT

| rex field=_raw "hard_bounced=\"+(?<hard>.*?)\"+" max_match=0 | rex field=_raw "hostname=\"+(?<hostname>.*?)\"+" max_match=0 | eval fields = mvzip(hard,hostname) 
| mvexpand fields 
| rex field=fields "(?<hard>\w+),(?<hostname>\w+)" 
| table _time hard hostname

Re: event contains same field with different values

Sukisen1981 — Fri, 15 Jun 2018 09:49:37 GMT

Finally- the above one should work..for some reason 'hard' and 'hostname' got missed out even after applying it within the code blocks, i dunno why !