Why is my table field from JSON not working on all fields?

[sboogaar]

We are working with the following JSON generated by a dcos/marathon api:

When I run:

index=dcos sourcetype="dcos:marathon:metrics" | table gauges.api.mesosphere.marathon.core.event.impl.stream.HttpEventStreamActorMetrics.number-of-streams.count

I get a nice table with all the expected numbers.

But, when I run:

index=dcos sourcetype="dcos:marathon:metrics" | table gauges.service.mesosphere.marathon.leaderDuration.count

All the fields are empty.

Why can I see the correct values for "gauges.api.mesosphere.marathon.core.event.impl.stream.HttpEventStreamActorMetrics.number-of-streams.count" But can not see it for gauges.service.mesosphere.marathon.leaderDuration.count
I also tried to get the data with spath like:

index=dcos sourcetype="dcos:marathon:metrics" | spath "gauges.service.mesosphere.marathon.leaderDuration.count" |  table  *

But again, the values are empty even though I can see gauges.service.mesosphere.marathon.leaderDuration.count in the table headings.

Even when I generate the searches with Splunk I get no data

[DalJeanis]

Splunk has a limitation on how big a json it is able to extract.

Let's verify that is not the issue. This should snip out all the nodes in the JSON before the leaderDuration node.

 index=dcos sourcetype="dcos:marathon:metrics" 
| head 1
| rex mode=sed field=_raw "s/(gauges:\s{)(.*)(service.mesosphere.marathon.leaderDuration)/\1\3/g"

Verify that that code kills the earlier data. After that, try

| table gauges.service.mesosphere.marathon.leaderDuration.count

and

|  spath "gauges.service.mesosphere.marathon.leaderDuration.count"

[kamlesh_vaghela]

@sboogaar
I have a doubt regarding below configurations. It might be hit in your event. Can you please reconfigure limits.conf if required and check again.

extraction_cutoff = <integer>
* For extract-all spath extraction mode, only apply extraction to the first
  <integer> number of bytes.
* Default: 5000

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bspath.5D

limit = <integer>
* The maximum number of fields that an automatic key-value field extraction
  (auto kv) can generate at search time.
* If search-time field extractions are disabled (KV_MODE=none in props.conf)
  then this setting determines the number of index-time fields that will be
  returned.
* The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype',
  'linecount', 'splunk_server', and 'splunk_server_group' do not count against
  this limit and will always be returned.
* Increase this setting if, for example, you have indexed data with a large
  number of columns and want to ensure that searches display all fields from
  the data.
* Default: 100

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bkv.5D

[sboogaar]

@kamlesh_vaghela changing the extraction_cutoff worked I did not need to update the limit. If you post it as an answer I will accept it.

[kamlesh_vaghela]

Great @sboogaar, extraction_cutoff worked for you.
Glad to help you.

[macadminrohit]

Does splunk create a field name gauges.service.mesosphere.marathon.leaderDuration.count similar to what it has created where it showed you the contents in the table.

[sboogaar]

@macadminrohit Yes see the last image.

[kamlesh_vaghela]

@sboogaar

Can you please share the sample JSON event??

[sboogaar]

@kamlesh_vaghela It is 34k characters long and contains private data so I can not share it, if you tell me what you want to check I will try to provide that information.

[woodcock]

If you cannot provide a sanitized event of identical size, then there is no good way for us to help.