Getting Data In

Is there JSON model validation in Splunk?

msrama5
Explorer

Hello, I have complex json being written to splunk and want to do model file validation , what is the best way to do this in splunk for each of the json data written to spunk ? apart from checking json matches model structure, want to check for mandatory values for some fields and format matching for some fields, can this be done inside splunk ?

{
"TestTransaction":{
"OrderEntryType":141,
"Number":69909696,
"CloseDate":"2020-02-03T15:31:38.1260000Z",
"ab":"test",
"Trans":[
{
"Amt":5.45,
"Desc":"test card",
"Id":"961071022758064128",
"Number":7777207236838910,
"ab":"test",
"$type":"test"
}
],
"TotalAmt":5.45,
"SubAmount":4.95,
"TaxAmount":0.5,
"DiscountAmount":0.0,
"Header":{
"ServiceType":null,
"RequestDate":"2020-02-03T15:31:38.1260000Z",
"$type":"Header"
},
"Preparation":"ConsOutOfStore",
"Details":{
"Discounts":[
],
"Items":[
{
"Qty":1.0,
"Sku":null,
"Price":4.45,
"Discounts":[
],
"OverrideDescription":null,
"OverridePrice":null,
"Suffix":null,
"ChildItems":[
{
"Qty":1.0,
"Sku":null,
"Price":0.0,
"Discounts":null,
"IsRefunded":false,
"IsTaxed":false,
"Summary":{
"TotalPrice":4.95,
"DiscountAmount":0,
"SubtotalAmount":4.95,
"$type":"testSummary"
},
"$type":"testItem"
}
],
"Taxes":[
{
"Name":"Sales Tax",
"Amount":50,
"$type":"testTax"
}
],
"ReceiptLines":[
],
"Delivery":null,
"$type":"testDetails"
},
"$type":"trans"
},
"RequestId":"test",
"MessageId":"test",
"$type":"testTransaction"
}
Tags (3)
0 Karma

manjunathmeti
Champion

You can create a new kvstore collection on search head and enforce type checking in collections.conf in an app.

collections.conf

[test]
enforceTypes = true
field.TestTransaction.OrderEntryType = number
field.TestTransaction.Number = number
field.TestTransaction.CloseDate = time

Then use REST API to write this data to collection, if field values in json doesn't match data type then insertion will fail and response is returned with an error.

curl -kv -u admin "https://localhost:8089/servicesNS/nobody/APP_NAME/storage/collections/data/test" -H 'Content-Type: application/json' -d '{"TestTransaction":{"OrderEntryType":"test","Number":69909696,"CloseDate":"2020-02-03T15:31:38.1260000Z"}}'

*<?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="ERROR">JSON in the request is invalid. (Failed to convert key='OrderEntryType' with value='test' to type '1')</msg>
  </messages>
</response>*
0 Karma

msrama5
Explorer

Thanks, I got the field validation, I also need to validate json data structure is not out of order (or) some fields missing and corrupted comparing to original json model template, we have seen this happen where a bad json with fields out of order would come at run time and need to alert by comparing with original model template and write requests that are corrupted, can the json data be compared with model template inside splunk ?

0 Karma

to4kawa
Ultra Champion
| makeresults 
| eval _raw="{
\"TestTransaction\":{
\"OrderEntryType\":141,
\"Number\":69909696,
\"CloseDate\":\"2020-02-03T15:31:38.1260000Z\",
\"ab\":\"test\",
\"Trans\":[
{
\"Amt\":5.45,
\"Desc\":\"test card\",
\"Id\":\"961071022758064128\",
\"Number\":7777207236838910,
\"ab\":\"test\",
\"$type\":\"test\"
}
],
\"TotalAmt\":5.45,
\"SubAmount\":4.95,
\"TaxAmount\":0.5,
\"DiscountAmount\":0.0,
\"Header\":{
\"ServiceType\":null,
\"RequestDate\":\"2020-02-03T15:31:38.1260000Z\",
\"$type\":\"Header\"
},
\"Preparation\":\"ConsOutOfStore\",
\"Details\":{
\"Discounts\":[
],
\"Items\":[
{
\"Qty\":1.0,
\"Sku\":null,
\"Price\":4.45,
\"Discounts\":[
],
\"OverrideDescription\":null,
\"OverridePrice\":null,
\"Suffix\":null,
\"ChildItems\":[
{
\"Qty\":1.0,
\"Sku\":null,
\"Price\":0.0,
\"Discounts\":null,
\"IsRefunded\":false,
\"IsTaxed\":false,
\"Summary\":{
\"TotalPrice\":4.95,
\"DiscountAmount\":0,
\"SubtotalAmount\":4.95,
\"$type\":\"testSummary\"
},
\"$type\":\"testItem\"
}
],
\"Taxes\":[
{
\"Name\":\"Sales Tax\",
\"Amount\":50,
\"$type\":\"testTax\"
}
],
\"ReceiptLines\":[
],
\"Delivery\":null,
\"$type\":\"testDetails\"
},
\"$type\":\"trans\"
},
\"RequestId\":\"test\",
\"MessageId\":\"test\",
\"$type\":\"testTransaction\"
}"
| spath

What's your expected result?
spath is enough?

0 Karma

msrama5
Explorer

I want to compare with json model file which has fieldnames and datatypes for each field , what I want to check is field names structures are aligned and data types mismatches don't exist comparing json model file with actual json data at run time and filter json requests which does match the model file

0 Karma
Get Updates on the Splunk Community!

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...