topic Re: What will the JSON Path for breaking this JSON be while ingesting it in Splunk ? in Getting Data In

What will the JSON Path for breaking this JSON be while ingesting it in Splunk ?

beingkaran — Wed, 30 Sep 2020 03:51:06 GMT

Please note that I want the JSON path expression and want to break this before ingesting it splunk and not to use spath after ingesting

JSON
[
{
"ticket_id":"423535",
},
{
"ticket_id":"422946",
},
{
"ticket_id":"272791",
},
{
"ticket_id":"240391",
},
]

Break it as

Event1:
{
"ticket_id":"423535",
}
Event 2:
{
"ticket_id":"422946",
}
Event 3:
{
"ticket_id":"272791",
}
Event 4:
{
"ticket_id":"240391",
}

Re: What will the JSON Path for breaking this JSON be while ingesting it in Splunk ?

kamlesh_vaghela — Wed, 29 Jan 2020 13:37:49 GMT

@beingkaran

Is it actual event you are getting ? if not then can you please share original event with masking confidential information.

Re: What will the JSON Path for breaking this JSON be while ingesting it in Splunk ?

beingkaran — Wed, 30 Sep 2020 03:53:14 GMT

Hi kamlesh this is the gist of it I can disclose more due to customer data but the structure is exact now I want to know the json path that will break them ticket by ticket

[
{
"ticket_id": "EN_3",
"circuit_id": "ENT3",
"customer_id": "C-1",
"prob_type": "LINKDOWN",
"severity": "Critical"
},
{
"ticket_id": "EN_4",
"circuit_id": "ENT4",
"customer_id": "C-2",
"prob_type": "LINKDOWN",
"severity": "Critical"
}
]

Re: What will the JSON Path for breaking this JSON be while ingesting it in Splunk ?

spavin — Wed, 30 Sep 2020 03:59:07 GMT

Hi @beingkaran,

If you on-board the JSON as-is, you can create search-time extractions to get all the fields, then you won't need to use spath to use them.

E.G. the following props.conf:

[ANSWERS_SEARCHTIME]
CHARSET=UTF-8
DATETIME_CONFIG=CURRENT
KV_MODE=JSON
LINE_BREAKER=\}(,?)
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
SEDCMD-firstbracket=s/^\[//
SEDCMD-lastbracket=s/\]$//
EXTRACT-ticket_id = "ticket_id"\s*:\s*"([^"]+)"
EXTRACT-circuit_id = "circuit_id"\s*:\s*"([^"]+)"
EXTRACT-customer_id = "customer_id"\s*:\s*"([^"]+)"
EXTRACT-prob_type = "prob_type"\s*:\s*"([^"]+)"
EXTRACT-severity = "severity"\s*:\s*"([^"]+)"

That will break the event up (assuming no nested brackets) and create search-time fields for all the keys. The props.conf will need to go on both the search head (for search-time extractions) and indexer (for the rest of the props config).

An alternative is to use indexed extractions. You can take advantage of the INDEXED_EXTRACTIONS setting:

[SPLUNK_ANSWERS]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = json
LINE_BREAKER = ([\r\n]+)

This results in all the fields being available, with no spath required:

Note that indexed extractions will take more space on indexers, and will need more CPU to initially on-board. You get the ability to use tstats though, which you can't do with search-time extractions.

Re: What will the JSON Path for breaking this JSON be while ingesting it in Splunk ?

to4kawa — Sat, 01 Feb 2020 03:44:37 GMT

KV_MODE=JSON

 EXTRACT-ticket_id = "ticket_id"\s*:\s*"([^"]+)"
 EXTRACT-circuit_id = "circuit_id"\s*:\s*"([^"]+)"
 EXTRACT-customer_id = "customer_id"\s*:\s*"([^"]+)"
 EXTRACT-prob_type = "prob_type"\s*:\s*"([^"]+)"
 EXTRACT-severity = "severity"\s*:\s*"([^"]+)"

Do you need above?

Re: What will the JSON Path for breaking this JSON be while ingesting it in Splunk ?

spavin — Sat, 01 Feb 2020 23:45:07 GMT

You can do either or both, depending on how you want the extracted fields to look.

KV_MODE will give extractions in the form of: {}.fieldname

EXTRACT- will give extractions in the form of: fieldname

Re: What will the JSON Path for breaking this JSON be while ingesting it in Splunk ?

to4kawa — Sun, 02 Feb 2020 03:55:19 GMT

Thanks @spavin

JSON_TRIM_BRACES_IN_ARRAY_NAMES = <boolean>
* Whether or not the JSON parser for 'INDEXED_EXTRACTIONS' strips curly
  braces from names of fields that are defined as arrays in JSON events.
* When the JSON parser extracts fields from JSON events, by default, it
  extracts array field names with the curly braces that indicate they
  are arrays ("{}") intact.
* For example, given the following partial JSON event:
    {"datetime":"08-20-2015 10:32:25.267 -0700","log_level":"INFO",...,
      data:{...,"fs_type":"ext4","mount_point":["/disk48","/disk22"],...}}

  Because the "mount_point" field in this event is an array of two
  values ("/disk48" and "/disk22"), the JSON parser sees the field as an
  array, and extracts it as such, including the braces that identify
  it as an array. The resulting field name is "data.mount_point{}").
* Set 'JSON_TRIM_BRACES_IN_ARRAY_NAMES' to "true" if you want the JSON
  parser to strip these curly braces from array field names. (In this
  example, the resulting field is instead "data.mount_point").
* CAUTION: Setting this to "true" makes array field names that are extracted
  at index time through the JSON parser inconsistent with search-time
  extraction of array field names through the 'spath' search command.
* Default: false

this option is useful, too.