How to extract multivalue fields and assigning them to dynamically named fields from same extraction using configuration instead of search time query?

[jmartens]

I can extract multi value fields from a field in events like these:

079184/Query key: ((0008,0016)) SOP Class UID [1.2.840.10008.5.1.4.1.1.481.3] | ((0008,0018)) SOP Instance UID [1.2.246.352.71.4.32219573436.547832.20200108135355] | ((0008,0052)) Query/Retrieve Level [IMAGE] For details see: \\REDACTED.log

And assign them to fieldnames extracted from the search results like this:

| makeresults | eval message="079184/Query key: ((0008,0016)) SOP Class UID [1.2.840.10008.5.1.4.1.1.481.3] | ((0008,0018)) SOP Instance UID [1.2.246.352.71.4.32219573436.547832.20200108135355] | ((0008,0052)) Query/Retrieve Level [IMAGE] For details see: \\REDACTED.log"
| rex field=message "(?<DICOMQueryKeys>\(\(.*])"
| makemv delim="|" DICOMQueryKeys
| mvexpand DICOMQueryKeys
| rex field=DICOMQueryKeys "\(\((?<DICOMQueryItem>(?<DICOMQueryFieldGroup>\w+),(?<DICOMQueryFieldElement>\w+))\)\)\s+(?<DICOMQueryFieldText>.*(?=\s+\[))\s+\[(?<DICOMQueryFieldValue>[^]]+)"
| eval DICOMQueryFieldText=replace(DICOMQueryFieldText,"\W","")
| eval {DICOMQueryFieldText}=DICOMQueryFieldValue
| fields - DICOMQueryKeys, DICOMQueryFieldValue
| table DICOMQueryFieldText, DICOMQueryItem, DICOMQueryFieldGroup, DICOMQueryFieldElement, QueryRetrieveLevel, SOPInstanceUID, SOPClassUID

So that I can use the textual labels as fields in Splunk see below:

I know how to register the first field extraction in the first line of above statements, but am unsure on how to add the subsequent statements to splunk so they are available to all users as fields when writing queries.

1. Where do I enter the makemv and mvexpand statements?
2. Should I use a series of subsequent transformations for the eval statements and subsequent regular expression field extractions?
3. How do I remove intermediate fields so users are not seeing them?

Any guidance and advice is much appreciated.

[to4kawa]

| makeresults 
| eval _raw="079184/Query key: ((0008,0016)) SOP Class UID [1.2.840.10008.5.1.4.1.1.481.3] | ((0008,0018)) SOP Instance UID [1.2.246.352.71.4.32219573436.547832.20200108135355] | ((0008,0052)) Query/Retrieve Level [IMAGE] For details see: \\REDACTED.log" 
    `comment("this is your log sample. from here, the logic")`
    `comment("Field extract, I decided to make it max_match=10, Please adjust")` 
| rex max_match=10 "(?<message>\(\((?<DICOMQueryItem>(?<DICOMQueryFieldGroup>\d+),(?<DICOMQueryFieldElement>\d+))\)\)\s+(?<DICOMQueryFieldText>.+?)\s+\[(?<DICOMQueryFieldValue>[^]]+)\])" 
    `comment("change multivalue event to single event")` 
| eval _counter=mvrange(0,mvcount(message)) 
| stats list(*) as * by _counter 
| foreach * 
    [ eval <<FIELD>> = mvindex(<<FIELD>>,_counter)] 
    `comment("Shaping, your logic")` 
| eval DICOMQueryFieldText=replace(DICOMQueryFieldText,"\W","") 
| eval {DICOMQueryFieldText}=DICOMQueryFieldValue 
| table DICOMQueryFieldText, DICOMQueryItem, DICOMQueryFieldGroup, DICOMQueryFieldElement, QueryRetrieveLevel, SOPInstanceUID, SOPClassUID 
| fields DICOMQueryFieldText, DICOMQueryItem, DICOMQueryFieldGroup, DICOMQueryFieldElement, QueryRetrieveLevel, SOPInstanceUID, SOPClassUID

Hi, @jmartens
If there is event( _raw ) , you can extract like this.

If the event is this chunk, there is no problem. will work.(ans Q1,Q2)

Q3 How do I remove intermediate fields so users are not seeing them?

Sorry, I am not sure. what's users ?
Do you need to control which fields are visible to users?

[acfecondo75]

Hey @jmartens,

I think your best bet for getting these fields and only the fields you specifically want would be to extract them from the logs via regex in your props.conf configuration.

I suggest this because all these fields are just values directly extracted from the logs (generally speaking, if the data comes directly from the log use an extraction. Use calculated fields or lookups for enriching the data that's found directly in the logs with data that isn't)

If you can determine which app is responsible for parsing these logs, you should add the configuration to that apps props.conf should be in a directory like this:

$SPLUNK_HOME/etc/apps/THIS_APP/local/props.conf

then find the stanza for that sourcetype SOURCETYPE_NAME
and add an EXTRACT:

EXTRACT-new_fields_for_THIS_SOURCETYPE=Query\skey:\s\(\((?<DICOMQueryItem>(?<DICOMQueryFieldGroup>\d+)\,(?<DICOMQueryFieldElement>\d+))\)\)\sSOP\sClass\sUID\s\[(?<SOPClassUID>[^\]]+)\].+SOP\sInstance\sUID \[(?<SOPInstanceUID>[^\]]+).+Query\/Retrieve\sLevel\s\[(?<QueryRetrieveLevel>[^\]]+)

I included a regex that I think captures all the desired fields in the way that you want them, but I would recommend reviewing it to make sure it's right and optimizing it any way you can as I did it as a proof of concept and didn't spend any time trying to make it efficient