Splunk Search

How to extract a repeating field from an event

stephenlclarke
New Member

I have data that looks like this:

AA=value1,BB=value2,BB=value3,BB=value4
AA=value5,BB=value6,BB=value7
AA=value8,BB=value9
AA=value10,BB=value11,BB=value12

How do I extract all the BB fields for each AA field?

Tags (1)
0 Karma
1 Solution

lguinn2
Legend

The following will automatically extract all key-value pairs (AA=value and BB=value,etc). This will include the CC fields, too.

What makes this different that the automatic field extraction is that it allows for multiple values for any field. So if there is more than one AA or BB or CC in the same event, the field will have multiple values.

In props.conf

[yoursourcetypehere]
KV_MODE = none
REPORT-ref = extract_fields

In transforms.conf

[extract_fields]
MV_ADD = true
DELIMS = ",", "="

Finally, if you want to see if this works, just run a short search like this:

yoursearchhere
| table AA BB

View solution in original post

cpetterborg
SplunkTrust
SplunkTrust

With your data:

<your search> | rex "AA=(?P<aa>[^,]+),[^B]*BB=(?P<bb_1>[^,]+)(,[^B]*BB=(?P<bb_2>[^,]+)(,[^B]*BB=(?P<bb_3>[^,]+))?)?"

If you need to have a "B" in your values, this may not work. If you have more than 3 sets of BB entries, you will have to add to this. If you could have another value (like CC=) before a BB, then you will have to change it a little, too.

Multi value stuff doesn't seem to me to work as well for this, because there isn't a list, like BB=1,2,3;CC=5,6,7, so this seems to be a reasonable alternative.

0 Karma

lguinn2
Legend

The following will automatically extract all key-value pairs (AA=value and BB=value,etc). This will include the CC fields, too.

What makes this different that the automatic field extraction is that it allows for multiple values for any field. So if there is more than one AA or BB or CC in the same event, the field will have multiple values.

In props.conf

[yoursourcetypehere]
KV_MODE = none
REPORT-ref = extract_fields

In transforms.conf

[extract_fields]
MV_ADD = true
DELIMS = ",", "="

Finally, if you want to see if this works, just run a short search like this:

yoursearchhere
| table AA BB

stephenlclarke
New Member

If I implement this on our production splunk infrastructure will this 'break' any existing reports and dashboards?

0 Karma

lguinn2
Legend

Probably not. The transformation will extract all the same fields that the automatic extraction finds; it just adds the multi-value fields ability.

However, if you have reports or dashboards that use the BB and CC fields, they might give a different answer now. But they were probably giving a wrong answer before...

0 Karma

_jgpm_
Communicator

Do you know how to do this if they weren't in KV pairs?

0 Karma

stephenlclarke
New Member

Actually...

The data looks a bit more like this;

AA=value1,BB=value2,CC=value3,BB=value4,BB=value5
AA=value6,BB=value7,BB=value8
AA=value9,BB=value10,CC=value11
AA=value12,BB=value13,CC=value14,BB=value15,CC=value16

There are other fields (CC) that I am not interested in that are mixed up with the BB fields. There is always only one AA field per event. Each BB field may have an associated CC field.

I am only interested in the BB fields per AA

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...