I have data that looks like this:
AA=value1,BB=value2,BB=value3,BB=value4
AA=value5,BB=value6,BB=value7
AA=value8,BB=value9
AA=value10,BB=value11,BB=value12
How do I extract all the BB fields for each AA field?
The following will automatically extract all key-value pairs (AA=value and BB=value,etc). This will include the CC fields, too.
What makes this different that the automatic field extraction is that it allows for multiple values for any field. So if there is more than one AA or BB or CC in the same event, the field will have multiple values.
In props.conf
[yoursourcetypehere]
KV_MODE = none
REPORT-ref = extract_fields
In transforms.conf
[extract_fields]
MV_ADD = true
DELIMS = ",", "="
Finally, if you want to see if this works, just run a short search like this:
yoursearchhere
| table AA BB
With your data:
<your search> | rex "AA=(?P<aa>[^,]+),[^B]*BB=(?P<bb_1>[^,]+)(,[^B]*BB=(?P<bb_2>[^,]+)(,[^B]*BB=(?P<bb_3>[^,]+))?)?"
If you need to have a "B" in your values, this may not work. If you have more than 3 sets of BB entries, you will have to add to this. If you could have another value (like CC=) before a BB, then you will have to change it a little, too.
Multi value stuff doesn't seem to me to work as well for this, because there isn't a list, like BB=1,2,3;CC=5,6,7
, so this seems to be a reasonable alternative.
The following will automatically extract all key-value pairs (AA=value and BB=value,etc). This will include the CC fields, too.
What makes this different that the automatic field extraction is that it allows for multiple values for any field. So if there is more than one AA or BB or CC in the same event, the field will have multiple values.
In props.conf
[yoursourcetypehere]
KV_MODE = none
REPORT-ref = extract_fields
In transforms.conf
[extract_fields]
MV_ADD = true
DELIMS = ",", "="
Finally, if you want to see if this works, just run a short search like this:
yoursearchhere
| table AA BB
If I implement this on our production splunk infrastructure will this 'break' any existing reports and dashboards?
Probably not. The transformation will extract all the same fields that the automatic extraction finds; it just adds the multi-value fields ability.
However, if you have reports or dashboards that use the BB and CC fields, they might give a different answer now. But they were probably giving a wrong answer before...
Do you know how to do this if they weren't in KV pairs?
Actually...
The data looks a bit more like this;
AA=value1,BB=value2,CC=value3,BB=value4,BB=value5
AA=value6,BB=value7,BB=value8
AA=value9,BB=value10,CC=value11
AA=value12,BB=value13,CC=value14,BB=value15,CC=value16
There are other fields (CC) that I am not interested in that are mixed up with the BB fields. There is always only one AA field per event. Each BB field may have an associated CC field.
I am only interested in the BB fields per AA