Getting Data In

Is there JSON model validation in Splunk?

msrama5
Explorer

Hello, I have complex json being written to splunk and want to do model file validation , what is the best way to do this in splunk for each of the json data written to spunk ? apart from checking json matches model structure, want to check for mandatory values for some fields and format matching for some fields, can this be done inside splunk ?

{
"TestTransaction":{
"OrderEntryType":141,
"Number":69909696,
"CloseDate":"2020-02-03T15:31:38.1260000Z",
"ab":"test",
"Trans":[
{
"Amt":5.45,
"Desc":"test card",
"Id":"961071022758064128",
"Number":7777207236838910,
"ab":"test",
"$type":"test"
}
],
"TotalAmt":5.45,
"SubAmount":4.95,
"TaxAmount":0.5,
"DiscountAmount":0.0,
"Header":{
"ServiceType":null,
"RequestDate":"2020-02-03T15:31:38.1260000Z",
"$type":"Header"
},
"Preparation":"ConsOutOfStore",
"Details":{
"Discounts":[
],
"Items":[
{
"Qty":1.0,
"Sku":null,
"Price":4.45,
"Discounts":[
],
"OverrideDescription":null,
"OverridePrice":null,
"Suffix":null,
"ChildItems":[
{
"Qty":1.0,
"Sku":null,
"Price":0.0,
"Discounts":null,
"IsRefunded":false,
"IsTaxed":false,
"Summary":{
"TotalPrice":4.95,
"DiscountAmount":0,
"SubtotalAmount":4.95,
"$type":"testSummary"
},
"$type":"testItem"
}
],
"Taxes":[
{
"Name":"Sales Tax",
"Amount":50,
"$type":"testTax"
}
],
"ReceiptLines":[
],
"Delivery":null,
"$type":"testDetails"
},
"$type":"trans"
},
"RequestId":"test",
"MessageId":"test",
"$type":"testTransaction"
}
Tags (3)
0 Karma

manjunathmeti
Champion

You can create a new kvstore collection on search head and enforce type checking in collections.conf in an app.

collections.conf

[test]
enforceTypes = true
field.TestTransaction.OrderEntryType = number
field.TestTransaction.Number = number
field.TestTransaction.CloseDate = time

Then use REST API to write this data to collection, if field values in json doesn't match data type then insertion will fail and response is returned with an error.

curl -kv -u admin "https://localhost:8089/servicesNS/nobody/APP_NAME/storage/collections/data/test" -H 'Content-Type: application/json' -d '{"TestTransaction":{"OrderEntryType":"test","Number":69909696,"CloseDate":"2020-02-03T15:31:38.1260000Z"}}'

*<?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="ERROR">JSON in the request is invalid. (Failed to convert key='OrderEntryType' with value='test' to type '1')</msg>
  </messages>
</response>*
0 Karma

msrama5
Explorer

Thanks, I got the field validation, I also need to validate json data structure is not out of order (or) some fields missing and corrupted comparing to original json model template, we have seen this happen where a bad json with fields out of order would come at run time and need to alert by comparing with original model template and write requests that are corrupted, can the json data be compared with model template inside splunk ?

0 Karma

to4kawa
Ultra Champion
| makeresults 
| eval _raw="{
\"TestTransaction\":{
\"OrderEntryType\":141,
\"Number\":69909696,
\"CloseDate\":\"2020-02-03T15:31:38.1260000Z\",
\"ab\":\"test\",
\"Trans\":[
{
\"Amt\":5.45,
\"Desc\":\"test card\",
\"Id\":\"961071022758064128\",
\"Number\":7777207236838910,
\"ab\":\"test\",
\"$type\":\"test\"
}
],
\"TotalAmt\":5.45,
\"SubAmount\":4.95,
\"TaxAmount\":0.5,
\"DiscountAmount\":0.0,
\"Header\":{
\"ServiceType\":null,
\"RequestDate\":\"2020-02-03T15:31:38.1260000Z\",
\"$type\":\"Header\"
},
\"Preparation\":\"ConsOutOfStore\",
\"Details\":{
\"Discounts\":[
],
\"Items\":[
{
\"Qty\":1.0,
\"Sku\":null,
\"Price\":4.45,
\"Discounts\":[
],
\"OverrideDescription\":null,
\"OverridePrice\":null,
\"Suffix\":null,
\"ChildItems\":[
{
\"Qty\":1.0,
\"Sku\":null,
\"Price\":0.0,
\"Discounts\":null,
\"IsRefunded\":false,
\"IsTaxed\":false,
\"Summary\":{
\"TotalPrice\":4.95,
\"DiscountAmount\":0,
\"SubtotalAmount\":4.95,
\"$type\":\"testSummary\"
},
\"$type\":\"testItem\"
}
],
\"Taxes\":[
{
\"Name\":\"Sales Tax\",
\"Amount\":50,
\"$type\":\"testTax\"
}
],
\"ReceiptLines\":[
],
\"Delivery\":null,
\"$type\":\"testDetails\"
},
\"$type\":\"trans\"
},
\"RequestId\":\"test\",
\"MessageId\":\"test\",
\"$type\":\"testTransaction\"
}"
| spath

What's your expected result?
spath is enough?

0 Karma

msrama5
Explorer

I want to compare with json model file which has fieldnames and datatypes for each field , what I want to check is field names structures are aligned and data types mismatches don't exist comparing json model file with actual json data at run time and filter json requests which does match the model file

0 Karma
Get Updates on the Splunk Community!

Updated Data Type Articles, Anniversary Celebrations, and More on Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

A Prelude to .conf25: Your Guide to Splunk University

Heading to Boston this September for .conf25? Get a jumpstart by arriving a few days early for Splunk ...

4 Ways the Splunk Community Helps You Prepare for .conf25

.conf25 is right around the corner, and whether you’re a first-time attendee or a seasoned Splunker, the ...