topic Re: Json huge data - issue with breaking the individual events in Getting Data In

Json huge data - issue with breaking the individual events

Nadhiya_Dubai — Mon, 09 Dec 2019 08:26:50 GMT

"Global Users":[
    {
         "AP name":"T2-GF-WDN-ISP-079", 
         "Auth":null, 
         "Bssid":"94:b4:0f:04:51:f1", 
         "Current switch":"172.30.97.41", 
         "Essid":"#DXB Free WiFi", 
         "IP":"10.11.0.23", 
         "MAC":"68:e7:c2:5d:a1:ad", 
         "Name":null, 
         "Phy":"a-HT", 
         "Profile":"FreeWifi-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"Free-Wifi-user-Role", 
         "Type":"Linux", 
         "User Type":"WIRELESS" 
    },
    {
         "AP name":"T3-L2-FD07-WDN-OSP-109", 
         "Auth":null, 
         "Bssid":"40:e3:d6:23:3b:21", 
         "Current switch":"172.30.97.111", 
         "Essid":"#DXB Free WiFi", 
         "IP":"10.234.0.213", 
         "MAC":"fc:aa:b6:17:1a:a3", 
         "Name":null, 
         "Phy":"g-HT", 
         "Profile":"T3-FreeWifi-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"Free-Wifi-user-Role", 
         "Type":"Linux", 
         "User Type":"WIRELESS" 
    },
    {
         "AP name":"T3-L2-FD12-WDN-ISP-020", 
         "Auth":"802.1x", 
         "Bssid":"b4:5d:50:f8:57:e2", 
         "Current switch":"172.30.97.112", 
         "Essid":"tenantauth", 
         "IP":"10.235.197.85", 
         "MAC":"d4:e6:b7:94:39:95", 
         "Name":"torydxb@tenant", 
         "Phy":"g-HT", 
         "Profile":"TENANTAUTH-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"TENANTAUTH-user-Role", 
         "Type":"Android", 
         "User Type":"WIRELESS" 
    },
    {
         "AP name":"CB-GF-FD07-WDN-OSP-050", 
         "Auth":"802.1x", 
         "Bssid":"20:a6:cd:30:9a:22", 
         "Current switch":"172.30.97.112", 
         "Essid":"ahlan", 
         "IP":"10.211.2.144", 
         "MAC":"48:9d:d1:6d:8d:e9", 
         "Name":"GNSCDWC02", 
         "Phy":"g-HT", 
         "Profile":"T3-CB-Ahlan-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"Ahlan-User-Role", 
         "Type":"Linux", 
         "User Type":"WIRELESS" 
    }
],
"_data":[
    "Total entries = 14995" 
],
"_meta":[
    "IP", 
    "MAC", 
    "Name", 
    "Current switch", 
    "Role", 
    "Auth", 
    "AP name", 
    "Roaming", 
    "Essid", 
    "Bssid", 
    "Phy", 
    "Profile", 
    "Type", 
    "User Type" 
]

Above is my json data . Well i have trimmmed the events ,its so huge lines in millions for a single event .
I tried giving the sourcetype as _json but its not breaking my events .Kindly help .I always have trouble while the data is in json format . Looking for the right solution and the explanation . Kindly help

Re: Json huge data - issue with breaking the individual events

bowesmana — Mon, 09 Dec 2019 10:41:24 GMT

JSON auto extraction will only extract, I believe, the first 5000 bytes. You need to use spath on the elements of the data you need.

Re: Json huge data - issue with breaking the individual events

starcher — Mon, 09 Dec 2019 14:15:58 GMT

If JSON is that big you should use code outside of Splunk to parse it into reasonable events and send those in. Also when sending in properly formed JSON use kv_mode = JSON on your sourcetype definition in props.

Re: Json huge data - issue with breaking the individual events

to4kawa — Mon, 09 Dec 2019 15:36:54 GMT

| makeresults 
 | eval _raw="{\"Global Users\":[{\"AP name\":\"T2-GF-WDN-ISP-079\",\"Auth\":null,\"Bssid\":\"94:b4:0f:04:51:f1\",\"Current switch\":\"172.30.97.41\",\"Essid\":\"#DXB Free WiFi\",\"IP\":\"10.11.0.23\",\"MAC\":\"68:e7:c2:5d:a1:ad\",\"Name\":null,\"Phy\":\"a-HT\",\"Profile\":\"FreeWifi-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"Free-Wifi-user-Role\",\"Type\":\"Linux\",\"User Type\":\"WIRELESS\"},{\"AP name\":\"T3-L2-FD07-WDN-OSP-109\",\"Auth\":null,\"Bssid\":\"40:e3:d6:23:3b:21\",\"Current switch\":\"172.30.97.111\",\"Essid\":\"#DXB Free WiFi\",\"IP\":\"10.234.0.213\",\"MAC\":\"fc:aa:b6:17:1a:a3\",\"Name\":null,\"Phy\":\"g-HT\",\"Profile\":\"T3-FreeWifi-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"Free-Wifi-user-Role\",\"Type\":\"Linux\",\"User Type\":\"WIRELESS\"},{\"AP name\":\"T3-L2-FD12-WDN-ISP-020\",\"Auth\":\"802.1x\",\"Bssid\":\"b4:5d:50:f8:57:e2\",\"Current switch\":\"172.30.97.112\",\"Essid\":\"tenantauth\",\"IP\":\"10.235.197.85\",\"MAC\":\"d4:e6:b7:94:39:95\",\"Name\":\"torydxb@tenant\",\"Phy\":\"g-HT\",\"Profile\":\"TENANTAUTH-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"TENANTAUTH-user-Role\",\"Type\":\"Android\",\"User Type\":\"WIRELESS\"},{\"AP name\":\"CB-GF-FD07-WDN-OSP-050\",\"Auth\":\"802.1x\",\"Bssid\":\"20:a6:cd:30:9a:22\",\"Current switch\":\"172.30.97.112\",\"Essid\":\"ahlan\",\"IP\":\"10.211.2.144\",\"MAC\":\"48:9d:d1:6d:8d:e9\",\"Name\":\"GNSCDWC02\",\"Phy\":\"g-HT\",\"Profile\":\"T3-CB-Ahlan-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"Ahlan-User-Role\",\"Type\":\"Linux\",\"User Type\":\"WIRELESS\"}],\"_data\":[\"Total entries = 14995\"],\"_meta\":[\"IP\",\"MAC\",\"Name\",\"Current switch\",\"Role\",\"Auth\",\"AP name\",\"Roaming\",\"Essid\",\"Bssid\",\"Phy\",\"Profile\",\"Type\",\"User Type\"]}"
 | spath  
 | fields - _*
 | rename "Global Users"{}.* as *
 | rename data{} as _data, meta{} as _meta
 | mvexpand IP
 | rename IP as _IP
 | streamstats count
 | foreach *
    [eval <<FIELD>> = mvindex('<<FIELD>>', count - 1)]
| rename _IP as IP, _data as data, _meta as meta
| table IP MAC Name "Current switch" Role Auth  "AP name" Roaming Essid Bssid Phy Profile Type "User Type" data

At this level, you can normally spath .

Re: Json huge data - issue with breaking the individual events

FrankVl — Mon, 09 Dec 2019 15:56:00 GMT

Maybe start with explaining what the desired behavior would be, because that is not very clear from your question. Do you want each { "AP name"... } section in a separate event?

In general, I would concur with the answer from @starcher that this looks like something you want to pre-process and then send into splunk as individual events, rather than massive json structs.

Re: Json huge data - issue with breaking the individual events

to4kawa — Tue, 10 Dec 2019 11:06:19 GMT

"Total entries = 14995"

props.conf
LINE_BREAKER in single line printed JSON doc

I hope this can be done well.