Getting Data In
Highlighted

Is there JSON model validation in Splunk?

Loves-to-Learn

Hello, I have complex json being written to splunk and want to do model file validation , what is the best way to do this in splunk for each of the json data written to spunk ? apart from checking json matches model structure, want to check for mandatory values for some fields and format matching for some fields, can this be done inside splunk ?

{
"TestTransaction":{
"OrderEntryType":141,
"Number":69909696,
"CloseDate":"2020-02-03T15:31:38.1260000Z",
"ab":"test",
"Trans":[
{
"Amt":5.45,
"Desc":"test card",
"Id":"961071022758064128",
"Number":7777207236838910,
"ab":"test",
"$type":"test"
}
],
"TotalAmt":5.45,
"SubAmount":4.95,
"TaxAmount":0.5,
"DiscountAmount":0.0,
"Header":{
"ServiceType":null,
"RequestDate":"2020-02-03T15:31:38.1260000Z",
"$type":"Header"
},
"Preparation":"ConsOutOfStore",
"Details":{
"Discounts":[
],
"Items":[
{
"Qty":1.0,
"Sku":null,
"Price":4.45,
"Discounts":[
],
"OverrideDescription":null,
"OverridePrice":null,
"Suffix":null,
"ChildItems":[
{
"Qty":1.0,
"Sku":null,
"Price":0.0,
"Discounts":null,
"IsRefunded":false,
"IsTaxed":false,
"Summary":{
"TotalPrice":4.95,
"DiscountAmount":0,
"SubtotalAmount":4.95,
"$type":"testSummary"
},
"$type":"testItem"
}
],
"Taxes":[
{
"Name":"Sales Tax",
"Amount":50,
"$type":"testTax"
}
],
"ReceiptLines":[
],
"Delivery":null,
"$type":"testDetails"
},
"$type":"trans"
},
"RequestId":"test",
"MessageId":"test",
"$type":"testTransaction"
}
Tags (3)
0 Karma
Highlighted

Re: Is there JSON model validation in Splunk?

Ultra Champion
| makeresults 
| eval _raw="{
\"TestTransaction\":{
\"OrderEntryType\":141,
\"Number\":69909696,
\"CloseDate\":\"2020-02-03T15:31:38.1260000Z\",
\"ab\":\"test\",
\"Trans\":[
{
\"Amt\":5.45,
\"Desc\":\"test card\",
\"Id\":\"961071022758064128\",
\"Number\":7777207236838910,
\"ab\":\"test\",
\"$type\":\"test\"
}
],
\"TotalAmt\":5.45,
\"SubAmount\":4.95,
\"TaxAmount\":0.5,
\"DiscountAmount\":0.0,
\"Header\":{
\"ServiceType\":null,
\"RequestDate\":\"2020-02-03T15:31:38.1260000Z\",
\"$type\":\"Header\"
},
\"Preparation\":\"ConsOutOfStore\",
\"Details\":{
\"Discounts\":[
],
\"Items\":[
{
\"Qty\":1.0,
\"Sku\":null,
\"Price\":4.45,
\"Discounts\":[
],
\"OverrideDescription\":null,
\"OverridePrice\":null,
\"Suffix\":null,
\"ChildItems\":[
{
\"Qty\":1.0,
\"Sku\":null,
\"Price\":0.0,
\"Discounts\":null,
\"IsRefunded\":false,
\"IsTaxed\":false,
\"Summary\":{
\"TotalPrice\":4.95,
\"DiscountAmount\":0,
\"SubtotalAmount\":4.95,
\"$type\":\"testSummary\"
},
\"$type\":\"testItem\"
}
],
\"Taxes\":[
{
\"Name\":\"Sales Tax\",
\"Amount\":50,
\"$type\":\"testTax\"
}
],
\"ReceiptLines\":[
],
\"Delivery\":null,
\"$type\":\"testDetails\"
},
\"$type\":\"trans\"
},
\"RequestId\":\"test\",
\"MessageId\":\"test\",
\"$type\":\"testTransaction\"
}"
| spath

What's your expected result?
spath is enough?

0 Karma
Highlighted

Re: Is there JSON model validation in Splunk?

Loves-to-Learn

I want to compare with json model file which has fieldnames and datatypes for each field , what I want to check is field names structures are aligned and data types mismatches don't exist comparing json model file with actual json data at run time and filter json requests which does match the model file

0 Karma
Highlighted

Re: Is there JSON model validation in Splunk?

Influencer

You can create a new kvstore collection on search head and enforce type checking in collections.conf in an app.

collections.conf

[test]
enforceTypes = true
field.TestTransaction.OrderEntryType = number
field.TestTransaction.Number = number
field.TestTransaction.CloseDate = time

Then use REST API to write this data to collection, if field values in json doesn't match data type then insertion will fail and response is returned with an error.

curl -kv -u admin "https://localhost:8089/servicesNS/nobody/APP_NAME/storage/collections/data/test" -H 'Content-Type: application/json' -d '{"TestTransaction":{"OrderEntryType":"test","Number":69909696,"CloseDate":"2020-02-03T15:31:38.1260000Z"}}'

*<?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="ERROR">JSON in the request is invalid. (Failed to convert key='OrderEntryType' with value='test' to type '1')</msg>
  </messages>
</response>*
0 Karma
Highlighted

Re: Is there JSON model validation in Splunk?

Loves-to-Learn

Thanks, I got the field validation, I also need to validate json data structure is not out of order (or) some fields missing and corrupted comparing to original json model template, we have seen this happen where a bad json with fields out of order would come at run time and need to alert by comparing with original model template and write requests that are corrupted, can the json data be compared with model template inside splunk ?

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.