Configuring field extractions for multivalue nested JSON events

[sharad06]

Hi experts,

I am working with nested JSON events which look as follows:

{   [-] 
     compliance:  <compliance_state>    
     ctupdate:  hostinfo    
     host_properties:   {   [-] 
         hwi_computer:  [   [+] 
        ]   
         hwi_disk:  [   [-] 
        {   [-] 
             counteract_tag:     health_data    
             since:  1519929479 
             value:  Device ID: C:;Drive Type: Local Disk;Free Space (Megabytes): 93159.7578125;Size (Megabytes): 122776.99609375;Name: C:;Description: Local Fixed Disk;Media Type: Fixed hard disk media;File System: NTFS    
        }   
        {   [-] 
             counteract_tag:     health_data    
             since:  1519929479 
             value:  Device ID: A:;Drive Type: Removable Disk;Name: A:;Description: 3 1/2 Inch Floppy Drive;Media Type: 3 1/2-Inch Floppy Disk - 720 KB - 512 bytes/sector; 
        }   
        {   [-] 
             counteract_tag:     health_data    
             since:  1519929479 
             value:  Device ID: D:;Drive Type: Compact Disc;Name: D:;Description: CD-ROM Disc;Media Type: Removable media other than floppy;    
        }   
        ]   
         hwi_network_adapters:  [   [+] 
        ]   
         hwi_physical_memory:   [   [+] 
        ]   
         hwi_processor: [   [+] 
        ]   
    }   
     ip:     <ipv4_address> 
     ipv6:   <ipv6_address> 
     mac:    <mac_address>  
     nbtdomain:  <domain>   
     user:   <user>
}

The above event contains a host_property named 'hwi_disk' which contains another JSON array carrying disk details of an endpoint under the key named 'value'. This 'value' contains different sub-keys separated by ':' and ';'. So far, I've been writing search queries using regexes to create my dashboards. The search queries look like:

`get_index` `get_sourcetypes` `ct_hostinfo` hwi_disk
| rename host_properties.hwi_disk{}.value as hwi_disk_val
| mvexpand hwi_disk_val
| rex field=hwi_disk_val "Device ID:\s*(?<DeviceID>[^;^$]*).*Free Space \(Megabytes\):\s*(?<FreeSpace>[^;^$]*).*Size \(Megabytes\):\s*(?<Size>[^;^$]*).*"
| stats latest(FreeSpace) as "Disk Space Available (MB)" latest(Size) as "Capacity (MB)" by ip DeviceID
| sort limit=10 -"Disk Space Available (MB)"
| rename ip as IP DeviceID as "Device ID"

I would like to know if I can write field extractions for such a multi-value event using conf files? Some pointers on how such extractions can written would be great help!

Thanks.

[valiquet]

You can use KV_MODE=JSON on SH if your JSON syntax is valid.

[p_gurav]

Hi,

Can you try something like this in search:

  | extract pairdelim="\r\n" kvdelim=":"

Let me know what it gives.

[sharad06]

Hi p_gaurav,

Thanks for your reply. I used the following query:

`get_index` `get_sourcetypes` `ct_hostinfo` hwi_network_adapters
| rename host_properties.hwi_network_adapters{}.value as hwi_network_adapters_val
| mvexpand hwi_network_adapters_val
| extract pairdelim="\r\n" kvdelim=":"

But it doesn't extract any of the nested values. I can see why as well because I want to run this 'extract command' on 'hwi_network_adapters_val' field, but to my knowledge, I can't specify a field parameter with 'extract' command (like I can do with the 'rex' command).

Finally, the greater challenge is to figure out how will this field extraction config translate to props.conf and transforms.conf stanzas.

Thanks for your help.

[sharad06]

KV_MODE=JSON can not extract fields for nested JSON events.