Splunk Search

Can you help me make a query that would Fill in values in braces from data inside the JSON?

jatwell2
New Member

Here is the source data:

{
  "contextValues": [
    "10.1.1.1",
    "10",
    "testhost"
  ],
  "contextTypes": [
    "IP",
    "threshold",
    "hostname"
  ],
  "message": "Latency on {hostname} with IP {IP} is higher than {threshold} ms."
}

I need a query to fill out messages. Items in braces in messages need to match contextType, to determine index and to retrieve contextValue on the same index. There could be zero to 10 items in braces and you cannot guarantee the order/index for specific contextTypes as they change.

I've played around with spath to grab data and multi-value fields. I can figure out how to get the data from contextVales at the same index as a matching search from contextTypes, but I'm failing around the multiple replace. I can do the first replace, but not all the others.

My next thought was to use regex to find the first occurrence and do the mvfind on index 0 of msgfield match, then a regex for 2nd match, etc. But, I can't figure how to do that w/ regex. The replace in the search below does not work correctly (replaces all items in braces w/ same value).

There's gotta be a simpler way. I just need the messages field filled out w/ replaced values.

| spath output=msg path=message | spath output=cType path=contextTypes{} | spath output=cVal path=contextValues{} | rex max_match=25 field=msg "{(?<msgfield>.+?)}" | eval val=mvindex(cVal,mvfind(cType,mvindex(msgfield,0))) | eval msg=replace (msg,"{.*?}",mvindex(cVal,mvfind(cType,mvindex(msgfield,0)))) | table msgfield, message, msg, val
Tags (3)
0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@jatwell2

Can you please try this?

YOUR_SEARCH
| rename *{} as * 
| eval temp=mvzip(contextTypes,contextValues) 
| stats count by temp, message 
| eval contextTypes=mvindex(split(temp,","),0),contextValues=mvindex(split(temp,","),1) 
| eval my_{contextTypes}=contextValues 
| stats values(*) as * by message
| foreach my_* [eval message= replace(message,"{"."<<MATCHSTR>>"."}",<<FIELD>>)]
| table message

My Sample Search:

| makeresults 
| eval _raw="{
    \"contextValues\": [
        \"10.1.1.1\",
        \"10\",
        \"testhost\"
                ],
    \"contextTypes\": [
        \"IP\",
        \"threshold\",
        \"hostname\"
                ],
    \"message\": \"Latency on {hostname} with IP {IP} is higher than {threshold} ms.\"
                }
    " 
| kv 
| rename *{} as * 
| eval temp=mvzip(contextTypes,contextValues) 
| stats count by temp, message 
| eval contextTypes=mvindex(split(temp,","),0),contextValues=mvindex(split(temp,","),1) 
| eval my_{contextTypes}=contextValues 
| stats values(*) as * by message
| foreach my_* [eval message= replace(message,"{"."<<MATCHSTR>>"."}",<<FIELD>>)]
| table message

Thanks

mhoogcarspel_sp
Splunk Employee
Splunk Employee

Enhanced from above idea:

| makeresults count=10 
| eval _raw="{
     \"contextValues\": [
         \"10.1.1.1\",
         \"10\",
         \"testhost\"
                         ],
     \"contextTypes\": [
         \"IP\",
         \"threshold\",
         \"hostname\"
                         ],
     \"message\": \"Latency on {hostname} with IP {IP} is higher than {threshold} ms.\"
                         }
     " 
| kv 
| rename _raw AS orig_raw 
| rename *{} as * 
| eval temp=split("my,my,my,my,my,my,my,my,my,my", ",") 
| eval _raw=mvzip(temp,mvzip(contextTypes,contextValues,"="),"_") 
| kv 
| fields - _raw 
| foreach my_* 
    [ eval message= replace(message,"{"."<<MATCHSTR>>"."}",<<FIELD>>)] 
| fields _time message
0 Karma

sr2801
New Member

Hi, this still is not working.

Here are a couple of information items:
1. For 'contextTypes' and 'contextValues', there can be anywhere from 1 pair to 25+ pairs. For "| eval temp=split("my,my,my,my,my,my,my,my,my,my", ",")", there are only 10 "my". This exact alert actually has 12 "pairs".
2. There may be anywhere from 1 to 25+ alerts in a sampling period (i.e past 24 hours) (my example above just happened to have 10 results at that time)

Here is my current code:

index=util_prod source=nutanix:alerts alertTypeUuid=A1024
| kv
| rename raw AS orig_raw
| rename *{} as *
| eval temp=split("my,my,my,my,my,my,my,my,my,my", ",")
| eval _raw=mvzip(temp,mvzip(contextTypes,contextValues,"="),"
")
| kv
| fields - raw
| foreach my
* [eval message=replace(message,"{"."<>"."}","\""+<>+"\"")]
| fields message
| table message

Here is what "_raw" is for the first result (I skewed some of the content data):

{"lastOccurrenceTimeStampInUsecs": 1555645972077454, "contextTypes": ["ip_address", "vm_type", "reboot_timestamp_str", "service_vm_external_ip", "service_vm_id", "maintenance_mode", "reboot_timestamp_secs", "ncc_version", "nos_version", "node_uuid", "node_serial", "block_serial"], "createdTimeStampInUsecs": 1555645972077454, "acknowledgedByUsername": "", "contextValues": ["10.1.2.3", "CVM", "Fri Apr 19 03:44:00 2019", "10.1.2.3", "43", "false", "15556734544210", "3.5.3.1-7663af3f", "5.5.4", "83f67898-7322-4945-b29d-7200b899433586f", "OM173S043400296", "17SM76343320208"], "resolved": false, "detailedMessage": "", "resolvedTimeStampInUsecs": 0, "clusterUuid": "0005591a-8548-03cb-0000-000043000176d5", "acknowledgedTimeStampInUsecs": 0, "checkId": "0005591a-8548-03cb-0000-0043400000176d5::3028", "nodeUuid": "83f67898-7322-4945-b29d-7200b434899586f", "alertTitle": "{vm_type} {ip_address} rebooted", "resolvedByUsername": "", "alertTypeUuid": "A1024", "originatingClusterUuid": "0005591a-8548-03cb-0000-0000434000176d5", "impact": "kUnknown", "alertDetails": null, "id": "abb434d6874-a78f-4f87-b066-c641be4ec904", "entityIds": ["0005594341a-8548-03cb-0000-0000000176d5::43"], "entityTypes": ["host"], "severity": "kCritical", "serviceVMId": "0005591a-8548-03cb-004300-0000000176d5::43", "categories": ["SystemIndicator", "ControllerVM"], "possibleCauses": [], "message": "{vm_type} {ip_address} has been rebooted on {reboot_timestamp_str}.", "acknowledged": false, "autoResolved": false, "entityUuids": ["83f67898-7322-4945-b29d-7200b65899586f"]}

I even added "| head 1" and the new code still fails.

Thoughts?

0 Karma

sr2801
New Member

I have been working on this with '@jatwell2'. Here is my code:

index=util_prod source=nutanix:alerts alertTypeUuid=A1024
| head 1
| rename *{} as *
| eval UpdatedMessage=message
| eval temp=mvzip(contextTypes,contextValues)
| stats count by temp, UpdatedMessage, message, alertTypeUuid, clusterUuid, severity
| eval cTypes=mvindex(split(temp,","),0),cValues=mvindex(split(temp,","),1)
| eval my_{cTypes}=cValues
| stats values(*) as * by UpdatedMessage
| foreach my_* [eval UpdatedMessage=replace(UpdatedMessage,"{"."<<MATCHSTR>>"."}","\""+<<FIELD>>+"\"")]
| table message, UpdatedMessage, _time

Notice the 'head 1'? That's because if there are multiple results from the query, the above code fails with 'UpdatedMessage' ending up blank. I tried '| stats values(*) as * by _time' but that only worked a few times, not every.

How can the above query be adjusted for multiple results?

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@sr2801

Is there the same events shared in question OR different?? Can you please share a sample event?

0 Karma

sr2801
New Member

Hi, so Nutanix has over 500 unique potential alerts (meaning many "alertTypeUuid"). The idea is that we use just a few queries for all 500 unique alerts. In a 5 minute sampling there could anywhere from 0 to many unique or repeat alerts.

For "alertTypeUuid=A1024" I have 10 instances in the past 7 days. Here is the table that the above code produces (without 'head 1'). Note that only a single line is produced (rather than 10 lines as I need):

message                                                                                                                    UpdatedMessage          _time
------------------------------------------------------------------------------------------------------------------------------------------------------
{vm_type} {ip_address} has been rebooted on {reboot_timestamp_str}.     (blank)                              (blank)

With 'head 1' (I obscured the IP and don't care the time is blank):

message                                                                                                                    UpdatedMessage                                                                                                _time
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
{vm_type} {ip_address} has been rebooted on {reboot_timestamp_str}.     "CVM" "10.x.y.z" has been rebooted on "Thu Mar 21 05:20:00 2019".       (blank)

I need the table to list 10 entries all with the proper UpdatedMessage. Does this help?

0 Karma

jatwell2
New Member

That works for this specific example, but the items in braces in message changes from event to event.

The next event might have 1, 3 or 5 different items in braces to replace from contextType/Value.

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@jatwell2

I have updated my answer. Please try and let me know for further assistance.

0 Karma

sr2801
New Member

Hi, can you take a look at this post again? I am having some issues with the code.

Thanks,
Steve

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.