topic Re: Get the list of unique combination in Splunk Search

Get the list of unique combination

samdjava — Wed, 04 Aug 2021 11:08:57 GMT

I would like to find

1. all unique combination of actionKey, modelName, programName

2. only consider data if they have a confidence score > 70.00

Splunk Raw Log -

2021-08-04 07:35:39,069 INFO [boundedElastic-87] [traceId="a4d01423048aa5de"] Request{userId='6699249',channelWise='SOCIAL', cid='1627958668279-9a93682610ee1c700c7e5d4ad01e8c76207274', sid=b8d2a070-f404-11eb-9cf4-5d474ec9ecbc, mlrecopred=[{actionKey=search, confidenceScore=83.46, modelName=model_forrest, programName=sapbased}, {actionKey=shipping_and_delivery, confidenceScore=82.94, modelName=model_forrest, programName=sapbased}, {actionKey=inventory_check, confidenceScore=65.21, modelName=model_forrest, programName=sapbased}, {actionKey=search, confidenceScore=63.46, modelName=event_handler, programName=sapbased}, {actionKey=shipping_and_delivery, confidenceScore=55.45, modelName=event_handler, programName=sapbased}], interactionId=0d6b031fdddba957, uniqueId='ed064f15d49c70ea7f540f7fe2ed2b7083e6eef8760f645f05d6600ad1208c3d'}

Re: Get the list of unique combination

kamlesh_vaghela — Wed, 04 Aug 2021 11:25:37 GMT

@samdjava

To extract value from raw event use below search.

YOUR_SEARCH | rex field=_raw "actionKey=(?<actionKey>[^,]+),\sconfidenceScore=(?<confidenceScore>[^,]+),\smodelName=(?<modelName>[^,]+),\sprogramName=(?<programName>[^}]+)" max_match=0 | eval t=mvzip(mvzip(mvzip(actionKey,confidenceScore),modelName),programName) | mvexpand t | eval actionKey = mvindex(split(t,","),0) ,confidenceScore= mvindex(split(t,","),1),modelName= mvindex(split(t,","),2),programName= mvindex(split(t,","),3) | fields - t

My Sample Search :

| makeresults | eval _raw="Request{userId='6699249',channelWise='SOCIAL', cid='1627958668279-9a93682610ee1c700c7e5d4ad01e8c76207274', sid=b8d2a070-f404-11eb-9cf4-5d474ec9ecbc, mlrecopred=[{actionKey=search, confidenceScore=83.46, modelName=model_forrest, programName=sapbased}, {actionKey=shipping_and_delivery, confidenceScore=82.94, modelName=model_forrest, programName=sapbased}, {actionKey=inventory_check, confidenceScore=65.21, modelName=model_forrest, programName=sapbased}, {actionKey=search, confidenceScore=63.46, modelName=event_handler, programName=sapbased}, {actionKey=shipping_and_delivery, confidenceScore=55.45, modelName=event_handler, programName=sapbased}], interactionId=0d6b031fdddba957, uniqueId='ed064f15d49c70ea7f540f7fe2ed2b7083e6eef8760f645f05d6600ad1208c3d'}" | rex field=_raw "actionKey=(?<actionKey>[^,]+),\sconfidenceScore=(?<confidenceScore>[^,]+),\smodelName=(?<modelName>[^,]+),\sprogramName=(?<programName>[^}]+)" max_match=0 | eval t=mvzip(mvzip(mvzip(actionKey,confidenceScore),modelName),programName) | mvexpand t | eval actionKey = mvindex(split(t,","),0) ,confidenceScore= mvindex(split(t,","),1),modelName= mvindex(split(t,","),2),programName= mvindex(split(t,","),3) | fields - t | de

For Unique event you can use below search

| dedup actionKey, modelName, programName

Use below search for filter event

| where confidenceScore > 70.00

Re: Get the list of unique combination

ITWhisperer — Wed, 04 Aug 2021 11:27:01 GMT

| makeresults | eval _raw="2021-08-04 07:35:39,069 INFO [boundedElastic-87] [traceId=\"a4d01423048aa5de\"] Request{userId='6699249',channelWise='SOCIAL', cid='1627958668279-9a93682610ee1c700c7e5d4ad01e8c76207274', sid=b8d2a070-f404-11eb-9cf4-5d474ec9ecbc, mlrecopred=[{actionKey=search, confidenceScore=83.46, modelName=model_forrest, programName=sapbased}, {actionKey=shipping_and_delivery, confidenceScore=82.94, modelName=model_forrest, programName=sapbased}, {actionKey=inventory_check, confidenceScore=65.21, modelName=model_forrest, programName=sapbased}, {actionKey=search, confidenceScore=63.46, modelName=event_handler, programName=sapbased}, {actionKey=shipping_and_delivery, confidenceScore=55.45, modelName=event_handler, programName=sapbased}], interactionId=0d6b031fdddba957, uniqueId='ed064f15d49c70ea7f540f7fe2ed2b7083e6eef8760f645f05d6600ad1208c3d'}" | rex "mlrecopred=\[(?<mlrecopred>[^\]]+)" | rex max_match=0 field=mlrecopred "\{(?<keyvalues>[^\}]+)\}" | mvexpand keyvalues | eval raw=_raw, _raw=keyvalues | extract pairdelim="," kvdelim="=" | where confidenceScore > 70 | dedup actionKey modelName programName | table actionKey modelName programName

Re: Get the list of unique combination

samdjava — Wed, 04 Aug 2021 12:28:58 GMT

Thanks for helping out