topic Splunk is not showing full JSON data from Chef Inspec log file on search in Getting Data In

Splunk is not showing full JSON data from Chef Inspec log file on search

jayaprasad800 — Mon, 06 Jan 2020 01:10:53 GMT

I have a json log file generated from chef inspec tool that contains 1000+ lines of data. The log file looks something like below.

    {
      "plat": {
        "name": "redhat",
        "rel": "7.6",
        "target": "test"
      },
      "profiles": [
        {
          "name": "sys",
          "ver": "0.1.0",
          "sha256": "e49",
          "title": "xyx,
          "maintainer": "xxx",
          "sum": "x",
          "license": "xyx",
          "copyright": "xxx",
          "email": "xyz@com",
          "supports": [
            {
              "platform": "os"
            }
          ],
          "attr": [],
          "groups": [
            {
              "id": "controls/fs.rb",
              "controls": [
                "4 FS",
                "4 FS",
                "4 FS"
              ]
            },
            {
              "id": "controls/mo.rb",
              "controls": [
                "1 mo"
              ],
              "title": "mo"
            },
            {
              "id": "controls/grp.rb",
              "controls": [
                "3 grp"
              ]
            },
            {
              "id": "controls/net.rb",
              "controls": [
                "6 net"
              ]
            },
            {
              "id": "controls/m.rb",
              "controls": [
                "8 Mail"
              ]
            },
            {
              "id": "controls/p.rb",
              "controls": [
                "9 sn"
              ]
            },
            {
              "id": "controls/rh.rb",
              "controls": [
                "10 sys"
              ]
            },
            {
              "id": "controls/usr.rb",
              "controls": [
                "2  usr",
                "2  usr",
                "2  usr",
                "2 usr",
                "2  usr",
                "2  usr",
                "2  usr",
                "2  usr"
              ],
              "title": "usr& rp "
            },
            {
              "id": "controls/dns.rb",
              "controls": [
                "7 DNS "
              ]
            },
            {
              "id": "controls/sys.rb",
              "controls": [
                "11 Sys"
              ]
            },
            {
              "id": "controls/time.rb",
              "controls": [
                "5 ntp",
                "5 chr"
              ]
            }
          ],
          "controls": [
            {
              "id": "4 FS ",
              "title": " FS",
              "desc": " FS",
              "desc": [
                {
                  "label": "default",
                  "data": " FS"
                },
                {
                  "label": "rationale",
                  "data": " FS"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "control '4  FS ' do                 
              "source": {
                "line": 1,
                "ref": "test/controls/fs.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": " FS",
                  "run_time": 0.0,
                  "start_time": "2020-01-07T09:54:49+11:00"
                }
              ]
            },
            {
              "id": "4 FS",
              "title": " FS",
              "desc": " FS",
              "desc": [
                {
                  "label": "default",
                  "data": " FS"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "control ' FS",
              "source": {
                "line": 12,
                "ref": "test/controls/fs.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": " FS",
                  "run_time": 0.1,
                  "start_time": "2020-01-07T09:54:49+11:00"
                }
              ]
            },
            {
              "id": "4 FS",
              "title": " FS",
              "desc": " FS",
              "desc": [
                {
                  "label": "default",
                  "data": " FS"
                }
              ],
              "impact": 1,
              "refs": [
                {
                  "ref": []
                }
              ],
              "tags": {},
              "code": " FS",
              "source": {
                "line": 22,
                "ref": "test/controls/fs.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": " FS",
                  "run_time": 0.5,
                  "start_time": "2020-01-07T09:54:49+11:00"
                }
              ]
            },
            {
              "id": "1.1 mo",
              "title": "mo",
              "desc": "mo",
              "desc": [
                {
                  "label": "default",
                  "data": "mo"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "mo",
              "source": {
                "line": 3,
                "ref": "test/controls/mo.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "mo",
                  "run_time": 0.1,
                  "start_time": "2020-01-07T09:54:49+11:00"
                },
                {
                  "status": "passed",
                  "code_desc": " mo",
                  "run_time": 0.6,
                  "start_time": "2020-01-07T09:54:49+11:00"
                }
              ]
            },
            {
              "id": "3.1 grp",
              "title": "grp",
              "desc": "grp",
              "desc": [
                {
                  "label": "default",
                  "data": "grp"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "grp",
              "source: {
                "line": 1,
                "ref": "test/controls/grp.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "grp",
                  "run_time": 0.7,
                  "start_time": "2020-01-07T09:54:49+11:00"
                }
              ]
            },
            {
              "id": "6.1 net",
              "title": "ntw,
              "desc": "ntw",
              "desc": [
                {
                  "label": "default",
                  "data": "ntw"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "ntw",
              "source": {
                "line": 1,
                "ref": "test/controls/net.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "skipped",
                  "code_desc": "No-op",
                  "run_time": 1.5,
                  "start_time": "2020-01-07T09:54:49+11:00",
                  "resource": "No-op",
                  "message": "Skipped "
                }
              ]
            },
            {
              "id": "8.1 Ma ",
              "title": "Ma",
              "desc": "Ma",
              "desc": [
                {
                  "label": "default",
                  "data": "Ma"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "Ma",
              "source": {
                "line": 1,
                "ref": "test/controls/ma.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "Ma",
                  "run_time": 0.9,
                  "start_time": "2020-01-07T09:54:49+11:00"
                }
              ]
            },
            {
              "id": "9.1 snmp",
              "title": "snmp",
              "desc": "snmp",
              "desc": [
                {
                  "label": "default",
                  "data": "snmp"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "snmp",
              "source": {
                "line": 1,
                "ref": "test/controls/snmp.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "snmp",
                  "run_time": 0.6,
                  "start_time": "2020-01-07T09:54:49+11:00"
                }
              ]
            },
            {
              "id": "10.1 sys reg",
              "title": "Sys Reg",
              "desc": "Sys Reg",
              "desc": [
                {
                  "label": "default",
                  "data": "Sys Reg"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "Sys Reg",
              "source": {
                "line": 1,
                "ref": "test/controls/rh.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "Sys Reg",
                  "run_time": 0.2,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "2.1usr",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source": {
                "line": 3,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.7,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "2 usr ",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source": {
                "line": 13,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.7,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "2 usr",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source": {
                "line": 33,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.9,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "2 usr",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source": {
                "line": 43,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.4,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "2.2.1 usr  exp",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source": {
                "line": 55,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.1,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "2.2.2 usr  exp",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source": {
                "line": 71,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.7,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "2.2.3  usr",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr "
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source": {
                "line": 87,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.2,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": " usr",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source: {
                "line": 119,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.2,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "7.1 DNS",
              "title": "DNS ,
              "desc": "DNS ",
              "desc": [
                {
                  "label": "default",
                  "data": "DNS"
                },
                {
                  "label": "rationale",
                  "data": "test"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "DNS 
              "source": {
                "line": 1,
                "ref": "test/controls/dns.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "DNS ",
                  "run_time": 0.4,
                  "start_time": "2020-01-07T09:54:50+11:00"
                },
                {
                  "status": "passed",
                  "code_desc": "DNS ",
                  "run_time": 0.9,
                  "start_time": "2020-01-07T09:54:51+11:00"
                },
                {
                  "status": "passed",
                  "code_desc": "DNS ",
                  "run_time": 9.5,
                  "start_time": "2020-01-07T09:54:51+11:00"
                }
              ]
            },
            {
              "id": "11 Syslog",
              "title": "logging",
              "desc": "chk sys",
              "desc": [
                {
                  "label": "default",
                  "data": "chk sys"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "logging",
              "source": {
                "line": 1,
                "ref": "test/controls/sys.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "logging",
                  "run_time": 0.1,
                  "start_time": "2020-01-07T09:54:51+11:00"
                },
                {
                  "status": "passed",
                  "code_desc": "logging",
                  "run_time": 0.5,
                  "start_time": "2020-01-07T09:54:51+11:00"
                }
              ]
            },
            {
              "id": "5.1 ntp",
              "title": "ntp",
              "desc": " ntp",
              "desc": [
                {
                  "label": "default",
                  "data": "ntp
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "NTP",
              "source": {
                "line": 1,
                "ref": "test/controls/time.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "skipped",
                  "code_desc": "No-op",
                  "run_time": 7.6,
                  "start_time": "2020-01-07T09:54:51+11:00",
                  "resource": "No-op",
                  "message": "Skipped"
                }
              ]
            },
            {
              "id": "5.2 time",
              "title": "time ",
              "desc": "time",
              "desc": [
                {
                  "label": "default",
                  "data": "time"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "time",
              "source: {
                "line": 21,
                "ref": "test/controls/time.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "time",
                  "run_time": 0.6,
                  "start_time": "2020-01-07T09:54:51+11:00"
                },
                {
                  "status": "passed",
                  "code_desc": "time",
                  "run_time": 0.4,
                  "start_time": "2020-01-07T09:54:51+11:00"
                }
              ]
            }
          ],
          "status": "loaded"
        }
      ],
      "stat": {
        "duration": 1.8
      },
      "ver": "4.18.24"
    }

However Splunk is only showing the first 302 lines from the file and not the whole line. Screen Shot attached

I am using the default _json as the source type and my requirement is to display the result from the Inspec as a table shown below.

But as splunk is not showing the all the data from the log file, I am not able to display all the values as a table.

Can someone please help me to get all the events from the inspec log file..

Thanks!

If you need more info please let me know.

Re: Splunk is not showing full JSON data from Chef Inspec log file on search

MuS — Mon, 06 Jan 2020 02:43:09 GMT

Hi jayaprasad800,

you need to change the MAX_EVENTS option in props.conf for the sourcetype _json, by default it is using 256 lines:

MAX_EVENTS = <integer>
* The maximum number of input lines to add to any event.
* Splunk software breaks after it reads the specified number of lines.
* Default: 256

Hope this helps ...

cheers, MuS

Re: Splunk is not showing full JSON data from Chef Inspec log file on search

jayaprasad800 — Mon, 06 Jan 2020 04:29:15 GMT

Hi MuS,

Thank you for the reply.

I have changed the MAX_EVENTS value to 1500 and restarted the service. But still Splunk is showing only 302 Lines.

Below is my _json settings

Name                                   Value    
CHARSET                           UTF-8
DATETIME_CONFIG
INDEXED_EXTRACTIONS  json
KV_MODE                           none
LINE_BREAKER                   ([\r\n]+)
MAX_EVENTS                      1500
NO_BINARY_CHECK           true
SHOULD_LINEMERGE        true
TRUNCATE                          0
category                             Structured
description                        JavaScript Object Notation format. For more information, visit http://json.org/
disabled                             false
pulldown_type                  true

Kindly check and let me know if I am missing any other option.

Re: Splunk is not showing full JSON data from Chef Inspec log file on search

jayaprasad800 — Wed, 30 Sep 2020 03:31:44 GMT

Hi MuS,

Thank you for the reply.

I have changed the MAX_EVENTS value to 1500 and restarted the service. But still Splunk is showing only 302 Lines.

Below is my _json props settings

Name Value

CHARSET UTF-8
DATETIME_CONFIG
INDEXED_EXTRACTIONS json
KV_MODE none
LINE_BREAKER ([\r\n]+)
MAX_EVENTS 1500
NO_BINARY_CHECK true
SHOULD_LINEMERGE true
TRUNCATE 0
category Structured
description JavaScript Object Notation format. For more information, visit http://json.org/
disabled false
pulldown_type true

Re: Splunk is not showing full JSON data from Chef Inspec log file on search

MuS — Mon, 06 Jan 2020 19:46:35 GMT

Hi jayaprasad800,

your posted example does only contains 236 lines, so this not really helpful. Please post an original event that gets truncated in Splunk, remember to sanitise it before posting it eq not PII information in it please.

Where did you add the props.conf? It has to be on the first full Splunk instance or the parsing instance that receives the event to be applied.
Did you restart Splunk after the modifications in props.conf?

cheers, MuS

Re: Splunk is not showing full JSON data from Chef Inspec log file on search

jayaprasad800 — Tue, 07 Jan 2020 00:48:57 GMT

Hi MuS,
I did the modification through splunk webgui and I have restarted the service after making the changes. Also please find the full Inspec log file.
Thanks in advance.

Re: Splunk is not showing full JSON data from Chef Inspec log file on search

MuS — Tue, 07 Jan 2020 02:03:21 GMT

Hi,

the updated samples are not JSON, it just looks like JSON. There are duplicated fields like desc on line 140 & 141 or there is the line 122 which is not JSON format because it missed the closing ",

I'm not sure if this is the main problem or just one of many problems that you encounter here ¯\_(ツ)_/¯

Maybe you should jump on the Splunk community Slack channel #getting-data-in and if your not yet on that slack signup here http://splk.it/slack

cheers, MuS

Re: Splunk is not showing full JSON data from Chef Inspec log file on search

jayaprasad800 — Tue, 07 Jan 2020 03:29:06 GMT

Hi Mus,

The logs are in json format only, I have used JQ to format it.

Also I have tried forwarding both logs( the plain json file and the formatted file with jq), but still same issue. splunk is not showing the full log file.

Sure, I will check in the slack channel as well.

thank you.