topic Re: Is there JSON model validation in Splunk? in Getting Data In

Is there JSON model validation in Splunk?

msrama5 — Tue, 04 Feb 2020 00:58:58 GMT

Hello, I have complex json being written to splunk and want to do model file validation , what is the best way to do this in splunk for each of the json data written to spunk ? apart from checking json matches model structure, want to check for mandatory values for some fields and format matching for some fields, can this be done inside splunk ?

{
"TestTransaction":{
"OrderEntryType":141,
"Number":69909696,
"CloseDate":"2020-02-03T15:31:38.1260000Z",
"ab":"test",
"Trans":[
{
"Amt":5.45,
"Desc":"test card",
"Id":"961071022758064128",
"Number":7777207236838910,
"ab":"test",
"$type":"test"
}
],
"TotalAmt":5.45,
"SubAmount":4.95,
"TaxAmount":0.5,
"DiscountAmount":0.0,
"Header":{
"ServiceType":null,
"RequestDate":"2020-02-03T15:31:38.1260000Z",
"$type":"Header"
},
"Preparation":"ConsOutOfStore",
"Details":{
"Discounts":[
],
"Items":[
{
"Qty":1.0,
"Sku":null,
"Price":4.45,
"Discounts":[
],
"OverrideDescription":null,
"OverridePrice":null,
"Suffix":null,
"ChildItems":[
{
"Qty":1.0,
"Sku":null,
"Price":0.0,
"Discounts":null,
"IsRefunded":false,
"IsTaxed":false,
"Summary":{
"TotalPrice":4.95,
"DiscountAmount":0,
"SubtotalAmount":4.95,
"$type":"testSummary"
},
"$type":"testItem"
}
],
"Taxes":[
{
"Name":"Sales Tax",
"Amount":50,
"$type":"testTax"
}
],
"ReceiptLines":[
],
"Delivery":null,
"$type":"testDetails"
},
"$type":"trans"
},
"RequestId":"test",
"MessageId":"test",
"$type":"testTransaction"
}

Re: Is there JSON model validation in Splunk?

to4kawa — Tue, 04 Feb 2020 13:14:44 GMT

| makeresults 
| eval _raw="{
\"TestTransaction\":{
\"OrderEntryType\":141,
\"Number\":69909696,
\"CloseDate\":\"2020-02-03T15:31:38.1260000Z\",
\"ab\":\"test\",
\"Trans\":[
{
\"Amt\":5.45,
\"Desc\":\"test card\",
\"Id\":\"961071022758064128\",
\"Number\":7777207236838910,
\"ab\":\"test\",
\"$type\":\"test\"
}
],
\"TotalAmt\":5.45,
\"SubAmount\":4.95,
\"TaxAmount\":0.5,
\"DiscountAmount\":0.0,
\"Header\":{
\"ServiceType\":null,
\"RequestDate\":\"2020-02-03T15:31:38.1260000Z\",
\"$type\":\"Header\"
},
\"Preparation\":\"ConsOutOfStore\",
\"Details\":{
\"Discounts\":[
],
\"Items\":[
{
\"Qty\":1.0,
\"Sku\":null,
\"Price\":4.45,
\"Discounts\":[
],
\"OverrideDescription\":null,
\"OverridePrice\":null,
\"Suffix\":null,
\"ChildItems\":[
{
\"Qty\":1.0,
\"Sku\":null,
\"Price\":0.0,
\"Discounts\":null,
\"IsRefunded\":false,
\"IsTaxed\":false,
\"Summary\":{
\"TotalPrice\":4.95,
\"DiscountAmount\":0,
\"SubtotalAmount\":4.95,
\"$type\":\"testSummary\"
},
\"$type\":\"testItem\"
}
],
\"Taxes\":[
{
\"Name\":\"Sales Tax\",
\"Amount\":50,
\"$type\":\"testTax\"
}
],
\"ReceiptLines\":[
],
\"Delivery\":null,
\"$type\":\"testDetails\"
},
\"$type\":\"trans\"
},
\"RequestId\":\"test\",
\"MessageId\":\"test\",
\"$type\":\"testTransaction\"
}"
| spath

What's your expected result?
spath is enough?

Re: Is there JSON model validation in Splunk?

msrama5 — Tue, 11 Feb 2020 06:51:51 GMT

I want to compare with json model file which has fieldnames and datatypes for each field , what I want to check is field names structures are aligned and data types mismatches don't exist comparing json model file with actual json data at run time and filter json requests which does match the model file

Re: Is there JSON model validation in Splunk?

manjunathmeti — Tue, 11 Feb 2020 07:34:55 GMT

You can create a new kvstore collection on search head and enforce type checking in collections.conf in an app.

collections.conf

[test]
enforceTypes = true
field.TestTransaction.OrderEntryType = number
field.TestTransaction.Number = number
field.TestTransaction.CloseDate = time

Then use REST API to write this data to collection, if field values in json doesn't match data type then insertion will fail and response is returned with an error.

curl -kv -u admin "https://localhost:8089/servicesNS/nobody/APP_NAME/storage/collections/data/test" -H 'Content-Type: application/json' -d '{"TestTransaction":{"OrderEntryType":"test","Number":69909696,"CloseDate":"2020-02-03T15:31:38.1260000Z"}}'

*<?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="ERROR">JSON in the request is invalid. (Failed to convert key='OrderEntryType' with value='test' to type '1')</msg>
  </messages>
</response>*

Re: Is there JSON model validation in Splunk?

msrama5 — Wed, 12 Feb 2020 06:18:02 GMT

Thanks, I got the field validation, I also need to validate json data structure is not out of order (or) some fields missing and corrupted comparing to original json model template, we have seen this happen where a bad json with fields out of order would come at run time and need to alert by comparing with original model template and write requests that are corrupted, can the json data be compared with model template inside splunk ?