Is there JSON model validation in Splunk?

[msrama5]

Hello, I have complex json being written to splunk and want to do model file validation , what is the best way to do this in splunk for each of the json data written to spunk ? apart from checking json matches model structure, want to check for mandatory values for some fields and format matching for some fields, can this be done inside splunk ?

{
"TestTransaction":{
"OrderEntryType":141,
"Number":69909696,
"CloseDate":"2020-02-03T15:31:38.1260000Z",
"ab":"test",
"Trans":[
{
"Amt":5.45,
"Desc":"test card",
"Id":"961071022758064128",
"Number":7777207236838910,
"ab":"test",
"$type":"test"
}
],
"TotalAmt":5.45,
"SubAmount":4.95,
"TaxAmount":0.5,
"DiscountAmount":0.0,
"Header":{
"ServiceType":null,
"RequestDate":"2020-02-03T15:31:38.1260000Z",
"$type":"Header"
},
"Preparation":"ConsOutOfStore",
"Details":{
"Discounts":[
],
"Items":[
{
"Qty":1.0,
"Sku":null,
"Price":4.45,
"Discounts":[
],
"OverrideDescription":null,
"OverridePrice":null,
"Suffix":null,
"ChildItems":[
{
"Qty":1.0,
"Sku":null,
"Price":0.0,
"Discounts":null,
"IsRefunded":false,
"IsTaxed":false,
"Summary":{
"TotalPrice":4.95,
"DiscountAmount":0,
"SubtotalAmount":4.95,
"$type":"testSummary"
},
"$type":"testItem"
}
],
"Taxes":[
{
"Name":"Sales Tax",
"Amount":50,
"$type":"testTax"
}
],
"ReceiptLines":[
],
"Delivery":null,
"$type":"testDetails"
},
"$type":"trans"
},
"RequestId":"test",
"MessageId":"test",
"$type":"testTransaction"
}

[manjunathmeti]

You can create a new kvstore collection on search head and enforce type checking in collections.conf in an app.

collections.conf

[test]
enforceTypes = true
field.TestTransaction.OrderEntryType = number
field.TestTransaction.Number = number
field.TestTransaction.CloseDate = time

Then use REST API to write this data to collection, if field values in json doesn't match data type then insertion will fail and response is returned with an error.

curl -kv -u admin "https://localhost:8089/servicesNS/nobody/APP_NAME/storage/collections/data/test" -H 'Content-Type: application/json' -d '{"TestTransaction":{"OrderEntryType":"test","Number":69909696,"CloseDate":"2020-02-03T15:31:38.1260000Z"}}'

*<?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="ERROR">JSON in the request is invalid. (Failed to convert key='OrderEntryType' with value='test' to type '1')</msg>
  </messages>
</response>*

[msrama5]

Thanks, I got the field validation, I also need to validate json data structure is not out of order (or) some fields missing and corrupted comparing to original json model template, we have seen this happen where a bad json with fields out of order would come at run time and need to alert by comparing with original model template and write requests that are corrupted, can the json data be compared with model template inside splunk ?

[to4kawa]

| makeresults 
| eval _raw="{
\"TestTransaction\":{
\"OrderEntryType\":141,
\"Number\":69909696,
\"CloseDate\":\"2020-02-03T15:31:38.1260000Z\",
\"ab\":\"test\",
\"Trans\":[
{
\"Amt\":5.45,
\"Desc\":\"test card\",
\"Id\":\"961071022758064128\",
\"Number\":7777207236838910,
\"ab\":\"test\",
\"$type\":\"test\"
}
],
\"TotalAmt\":5.45,
\"SubAmount\":4.95,
\"TaxAmount\":0.5,
\"DiscountAmount\":0.0,
\"Header\":{
\"ServiceType\":null,
\"RequestDate\":\"2020-02-03T15:31:38.1260000Z\",
\"$type\":\"Header\"
},
\"Preparation\":\"ConsOutOfStore\",
\"Details\":{
\"Discounts\":[
],
\"Items\":[
{
\"Qty\":1.0,
\"Sku\":null,
\"Price\":4.45,
\"Discounts\":[
],
\"OverrideDescription\":null,
\"OverridePrice\":null,
\"Suffix\":null,
\"ChildItems\":[
{
\"Qty\":1.0,
\"Sku\":null,
\"Price\":0.0,
\"Discounts\":null,
\"IsRefunded\":false,
\"IsTaxed\":false,
\"Summary\":{
\"TotalPrice\":4.95,
\"DiscountAmount\":0,
\"SubtotalAmount\":4.95,
\"$type\":\"testSummary\"
},
\"$type\":\"testItem\"
}
],
\"Taxes\":[
{
\"Name\":\"Sales Tax\",
\"Amount\":50,
\"$type\":\"testTax\"
}
],
\"ReceiptLines\":[
],
\"Delivery\":null,
\"$type\":\"testDetails\"
},
\"$type\":\"trans\"
},
\"RequestId\":\"test\",
\"MessageId\":\"test\",
\"$type\":\"testTransaction\"
}"
| spath

What's your expected result?
spath is enough?

[msrama5]

I want to compare with json model file which has fieldnames and datatypes for each field , what I want to check is field names structures are aligned and data types mismatches don't exist comparing json model file with actual json data at run time and filter json requests which does match the model file