topic Re: Why is my table field from JSON not working on all fields? in Getting Data In

Why is my table field from JSON not working on all fields?

sboogaar — Tue, 18 Dec 2018 15:49:40 GMT

We are working with the following JSON generated by a dcos/marathon api:

When I run:

index=dcos sourcetype="dcos:marathon:metrics" | table gauges.api.mesosphere.marathon.core.event.impl.stream.HttpEventStreamActorMetrics.number-of-streams.count

I get a nice table with all the expected numbers.

But, when I run:

index=dcos sourcetype="dcos:marathon:metrics" | table gauges.service.mesosphere.marathon.leaderDuration.count

All the fields are empty.

Why can I see the correct values for "gauges.api.mesosphere.marathon.core.event.impl.stream.HttpEventStreamActorMetrics.number-of-streams.count" But can not see it for gauges.service.mesosphere.marathon.leaderDuration.count
I also tried to get the data with spath like:

index=dcos sourcetype="dcos:marathon:metrics" | spath "gauges.service.mesosphere.marathon.leaderDuration.count" |  table  *

But again, the values are empty even though I can see gauges.service.mesosphere.marathon.leaderDuration.count in the table headings.

Even when I generate the searches with Splunk I get no data

Re: Why is my table field from JSON not working on all fields?

kamlesh_vaghela — Tue, 18 Dec 2018 17:06:00 GMT

@sboogaar

Can you please share the sample JSON event??

Re: Why is my table field from JSON not working on all fields?

macadminrohit — Tue, 18 Dec 2018 17:21:34 GMT

Does splunk create a field name gauges.service.mesosphere.marathon.leaderDuration.count similar to what it has created where it showed you the contents in the table.

Re: Why is my table field from JSON not working on all fields?

sboogaar — Wed, 19 Dec 2018 07:55:46 GMT

@macadminrohit Yes see the last image.

Re: Why is my table field from JSON not working on all fields?

sboogaar — Wed, 19 Dec 2018 08:52:20 GMT

@kamlesh_vaghela It is 34k characters long and contains private data so I can not share it, if you tell me what you want to check I will try to provide that information.

Re: Why is my table field from JSON not working on all fields?

kamlesh_vaghela — Wed, 19 Dec 2018 10:13:45 GMT

@sboogaar
I have a doubt regarding below configurations. It might be hit in your event. Can you please reconfigure limits.conf if required and check again.

extraction_cutoff = <integer>
* For extract-all spath extraction mode, only apply extraction to the first
  <integer> number of bytes.
* Default: 5000

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bspath.5D

limit = <integer>
* The maximum number of fields that an automatic key-value field extraction
  (auto kv) can generate at search time.
* If search-time field extractions are disabled (KV_MODE=none in props.conf)
  then this setting determines the number of index-time fields that will be
  returned.
* The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype',
  'linecount', 'splunk_server', and 'splunk_server_group' do not count against
  this limit and will always be returned.
* Increase this setting if, for example, you have indexed data with a large
  number of columns and want to ensure that searches display all fields from
  the data.
* Default: 100

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bkv.5D

Re: Why is my table field from JSON not working on all fields?

DalJeanis — Wed, 02 Jan 2019 17:38:10 GMT

Splunk has a limitation on how big a json it is able to extract.

Let's verify that is not the issue. This should snip out all the nodes in the JSON before the leaderDuration node.

 index=dcos sourcetype="dcos:marathon:metrics" 
| head 1
| rex mode=sed field=_raw "s/(gauges:\s{)(.*)(service.mesosphere.marathon.leaderDuration)/\1\3/g"

Verify that that code kills the earlier data. After that, try

| table gauges.service.mesosphere.marathon.leaderDuration.count

and

|  spath "gauges.service.mesosphere.marathon.leaderDuration.count"

Re: Why is my table field from JSON not working on all fields?

woodcock — Thu, 03 Jan 2019 01:52:15 GMT

If you cannot provide a sanitized event of identical size, then there is no good way for us to help.

Re: Why is my table field from JSON not working on all fields?

sboogaar — Tue, 29 Sep 2020 22:38:37 GMT

@kamlesh_vaghela changing the extraction_cutoff worked I did not need to update the limit. If you post it as an answer I will accept it.

Re: Why is my table field from JSON not working on all fields?

kamlesh_vaghela — Thu, 03 Jan 2019 09:48:24 GMT

Great @sboogaar, extraction_cutoff worked for you.
Glad to help you.