Getting Data In
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question

Splunk is not showing full JSON data from Chef Inspec log file on search

jayaprasad800
Engager
‎01-05-2020 05:10 PM

I have a json log file generated from chef inspec tool that contains 1000+ lines of data. The log file looks something like below.

    {
      "plat": {
        "name": "redhat",
        "rel": "7.6",
        "target": "test"
      },
      "profiles": [
        {
          "name": "sys",
          "ver": "0.1.0",
          "sha256": "e49",
          "title": "xyx,
          "maintainer": "xxx",
          "sum": "x",
          "license": "xyx",
          "copyright": "xxx",
          "email": "xyz@com",
          "supports": [
            {
              "platform": "os"
            }
          ],
          "attr": [],
          "groups": [
            {
              "id": "controls/fs.rb",
              "controls": [
                "4 FS",
                "4 FS",
                "4 FS"
              ]
            },
            {
              "id": "controls/mo.rb",
              "controls": [
                "1 mo"
              ],
              "title": "mo"
            },
            {
              "id": "controls/grp.rb",
              "controls": [
                "3 grp"
              ]
            },
            {
              "id": "controls/net.rb",
              "controls": [
                "6 net"
              ]
            },
            {
              "id": "controls/m.rb",
              "controls": [
                "8 Mail"
              ]
            },
            {
              "id": "controls/p.rb",
              "controls": [
                "9 sn"
              ]
            },
            {
              "id": "controls/rh.rb",
              "controls": [
                "10 sys"
              ]
            },
            {
              "id": "controls/usr.rb",
              "controls": [
                "2  usr",
                "2  usr",
                "2  usr",
                "2 usr",
                "2  usr",
                "2  usr",
                "2  usr",
                "2  usr"
              ],
              "title": "usr& rp "
            },
            {
              "id": "controls/dns.rb",
              "controls": [
                "7 DNS "
              ]
            },
            {
              "id": "controls/sys.rb",
              "controls": [
                "11 Sys"
              ]
            },
            {
              "id": "controls/time.rb",
              "controls": [
                "5 ntp",
                "5 chr"
              ]
            }
          ],
          "controls": [
            {
              "id": "4 FS ",
              "title": " FS",
              "desc": " FS",
              "desc": [
                {
                  "label": "default",
                  "data": " FS"
                },
                {
                  "label": "rationale",
                  "data": " FS"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "control '4  FS ' do                 
              "source": {
                "line": 1,
                "ref": "test/controls/fs.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": " FS",
                  "run_time": 0.0,
                  "start_time": "2020-01-07T09:54:49+11:00"
                }
              ]
            },
            {
              "id": "4 FS",
              "title": " FS",
              "desc": " FS",
              "desc": [
                {
                  "label": "default",
                  "data": " FS"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "control ' FS",
              "source": {
                "line": 12,
                "ref": "test/controls/fs.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": " FS",
                  "run_time": 0.1,
                  "start_time": "2020-01-07T09:54:49+11:00"
                }
              ]
            },
            {
              "id": "4 FS",
              "title": " FS",
              "desc": " FS",
              "desc": [
                {
                  "label": "default",
                  "data": " FS"
                }
              ],
              "impact": 1,
              "refs": [
                {
                  "ref": []
                }
              ],
              "tags": {},
              "code": " FS",
              "source": {
                "line": 22,
                "ref": "test/controls/fs.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": " FS",
                  "run_time": 0.5,
                  "start_time": "2020-01-07T09:54:49+11:00"
                }
              ]
            },
            {
              "id": "1.1 mo",
              "title": "mo",
              "desc": "mo",
              "desc": [
                {
                  "label": "default",
                  "data": "mo"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "mo",
              "source": {
                "line": 3,
                "ref": "test/controls/mo.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "mo",
                  "run_time": 0.1,
                  "start_time": "2020-01-07T09:54:49+11:00"
                },
                {
                  "status": "passed",
                  "code_desc": " mo",
                  "run_time": 0.6,
                  "start_time": "2020-01-07T09:54:49+11:00"
                }
              ]
            },
            {
              "id": "3.1 grp",
              "title": "grp",
              "desc": "grp",
              "desc": [
                {
                  "label": "default",
                  "data": "grp"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "grp",
              "source: {
                "line": 1,
                "ref": "test/controls/grp.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "grp",
                  "run_time": 0.7,
                  "start_time": "2020-01-07T09:54:49+11:00"
                }
              ]
            },
            {
              "id": "6.1 net",
              "title": "ntw,
              "desc": "ntw",
              "desc": [
                {
                  "label": "default",
                  "data": "ntw"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "ntw",
              "source": {
                "line": 1,
                "ref": "test/controls/net.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "skipped",
                  "code_desc": "No-op",
                  "run_time": 1.5,
                  "start_time": "2020-01-07T09:54:49+11:00",
                  "resource": "No-op",
                  "message": "Skipped "
                }
              ]
            },
            {
              "id": "8.1 Ma ",
              "title": "Ma",
              "desc": "Ma",
              "desc": [
                {
                  "label": "default",
                  "data": "Ma"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "Ma",
              "source": {
                "line": 1,
                "ref": "test/controls/ma.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "Ma",
                  "run_time": 0.9,
                  "start_time": "2020-01-07T09:54:49+11:00"
                }
              ]
            },
            {
              "id": "9.1 snmp",
              "title": "snmp",
              "desc": "snmp",
              "desc": [
                {
                  "label": "default",
                  "data": "snmp"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "snmp",
              "source": {
                "line": 1,
                "ref": "test/controls/snmp.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "snmp",
                  "run_time": 0.6,
                  "start_time": "2020-01-07T09:54:49+11:00"
                }
              ]
            },
            {
              "id": "10.1 sys reg",
              "title": "Sys Reg",
              "desc": "Sys Reg",
              "desc": [
                {
                  "label": "default",
                  "data": "Sys Reg"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "Sys Reg",
              "source": {
                "line": 1,
                "ref": "test/controls/rh.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "Sys Reg",
                  "run_time": 0.2,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "2.1usr",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source": {
                "line": 3,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.7,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "2 usr ",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source": {
                "line": 13,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.7,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "2 usr",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source": {
                "line": 33,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.9,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "2 usr",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source": {
                "line": 43,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.4,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "2.2.1 usr  exp",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source": {
                "line": 55,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.1,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "2.2.2 usr  exp",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source": {
                "line": 71,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.7,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "2.2.3  usr",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr "
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source": {
                "line": 87,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.2,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": " usr",
              "title": "usr",
              "desc": "usr",
              "desc": [
                {
                  "label": "default",
                  "data": "usr"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "usr",
              "source: {
                "line": 119,
                "ref": "test/controls/usr.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "usr",
                  "run_time": 0.2,
                  "start_time": "2020-01-07T09:54:50+11:00"
                }
              ]
            },
            {
              "id": "7.1 DNS",
              "title": "DNS ,
              "desc": "DNS ",
              "desc": [
                {
                  "label": "default",
                  "data": "DNS"
                },
                {
                  "label": "rationale",
                  "data": "test"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "DNS 
              "source": {
                "line": 1,
                "ref": "test/controls/dns.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "DNS ",
                  "run_time": 0.4,
                  "start_time": "2020-01-07T09:54:50+11:00"
                },
                {
                  "status": "passed",
                  "code_desc": "DNS ",
                  "run_time": 0.9,
                  "start_time": "2020-01-07T09:54:51+11:00"
                },
                {
                  "status": "passed",
                  "code_desc": "DNS ",
                  "run_time": 9.5,
                  "start_time": "2020-01-07T09:54:51+11:00"
                }
              ]
            },
            {
              "id": "11 Syslog",
              "title": "logging",
              "desc": "chk sys",
              "desc": [
                {
                  "label": "default",
                  "data": "chk sys"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "logging",
              "source": {
                "line": 1,
                "ref": "test/controls/sys.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "logging",
                  "run_time": 0.1,
                  "start_time": "2020-01-07T09:54:51+11:00"
                },
                {
                  "status": "passed",
                  "code_desc": "logging",
                  "run_time": 0.5,
                  "start_time": "2020-01-07T09:54:51+11:00"
                }
              ]
            },
            {
              "id": "5.1 ntp",
              "title": "ntp",
              "desc": " ntp",
              "desc": [
                {
                  "label": "default",
                  "data": "ntp
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "NTP",
              "source": {
                "line": 1,
                "ref": "test/controls/time.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "skipped",
                  "code_desc": "No-op",
                  "run_time": 7.6,
                  "start_time": "2020-01-07T09:54:51+11:00",
                  "resource": "No-op",
                  "message": "Skipped"
                }
              ]
            },
            {
              "id": "5.2 time",
              "title": "time ",
              "desc": "time",
              "desc": [
                {
                  "label": "default",
                  "data": "time"
                }
              ],
              "impact": 1,
              "refs": [],
              "tags": {},
              "code": "time",
              "source: {
                "line": 21,
                "ref": "test/controls/time.rb"
              },
              "data": {},
              "results": [
                {
                  "status": "passed",
                  "code_desc": "time",
                  "run_time": 0.6,
                  "start_time": "2020-01-07T09:54:51+11:00"
                },
                {
                  "status": "passed",
                  "code_desc": "time",
                  "run_time": 0.4,
                  "start_time": "2020-01-07T09:54:51+11:00"
                }
              ]
            }
          ],
          "status": "loaded"
        }
      ],
      "stat": {
        "duration": 1.8
      },
      "ver": "4.18.24"
    }

However Splunk is only showing the first 302 lines from the file and not the whole line. Screen Shot attached

alt text

I am using the default _json as the source type and my requirement is to display the result from the Inspec as a table shown below.

alt text

But as splunk is not showing the all the data from the log file, I am not able to display all the values as a table.

Can someone please help me to get all the events from the inspec log file..

Thanks!

If you need more info please let me know.

Tags (1)
Preview file
1 KB
Preview file
1 KB
0 Karma
Reply

jayaprasad800
Engager
‎01-05-2020 08:29 PM

Hi MuS,

Thank you for the reply.

I have changed the MAX_EVENTS value to 1500 and restarted the service. But still Splunk is showing only 302 Lines.

Below is my _json settings

Name                                   Value    
CHARSET                           UTF-8
DATETIME_CONFIG
INDEXED_EXTRACTIONS  json
KV_MODE                           none
LINE_BREAKER                   ([\r\n]+)
MAX_EVENTS                      1500
NO_BINARY_CHECK           true
SHOULD_LINEMERGE        true
TRUNCATE                          0
category                             Structured
description                        JavaScript Object Notation format. For more information, visit http://json.org/
disabled                             false
pulldown_type                  true

Kindly check and let me know if I am missing any other option.

0 Karma
Reply

MuS
Legend
‎01-05-2020 06:43 PM

Hi jayaprasad800,

you need to change the MAX_EVENTS option in props.conf for the sourcetype _json, by default it is using 256 lines:

MAX_EVENTS = <integer>
* The maximum number of input lines to add to any event.
* Splunk software breaks after it reads the specified number of lines.
* Default: 256

Hope this helps ...

cheers, MuS

0 Karma
Reply

jayaprasad800
Engager
‎01-05-2020 10:02 PM

Hi MuS,

Thank you for the reply.

I have changed the MAX_EVENTS value to 1500 and restarted the service. But still Splunk is showing only 302 Lines.

Below is my _json props settings

Name Value

  1. CHARSET UTF-8
  2. DATETIME_CONFIG
  3. INDEXED_EXTRACTIONS json
  4. KV_MODE none
  5. LINE_BREAKER ([\r\n]+)
  6. MAX_EVENTS 1500
  7. NO_BINARY_CHECK true
  8. SHOULD_LINEMERGE true
  9. TRUNCATE 0
  10. category Structured
  11. description JavaScript Object Notation format. For more information, visit http://json.org/
  12. disabled false
  13. pulldown_type true
0 Karma
Reply

MuS
Legend
‎01-06-2020 11:46 AM

Hi jayaprasad800,

your posted example does only contains 236 lines, so this not really helpful. Please post an original event that gets truncated in Splunk, remember to sanitise it before posting it eq not PII information in it please.

Where did you add the props.conf? It has to be on the first full Splunk instance or the parsing instance that receives the event to be applied.
Did you restart Splunk after the modifications in props.conf?

cheers, MuS

0 Karma
Reply

jayaprasad800
Engager
‎01-06-2020 04:48 PM

Hi MuS,
I did the modification through splunk webgui and I have restarted the service after making the changes. Also please find the full Inspec log file.
Thanks in advance.

0 Karma
Reply

MuS
Legend
‎01-06-2020 06:03 PM

Hi,

the updated samples are not JSON, it just looks like JSON. There are duplicated fields like desc on line 140 & 141 or there is the line 122 which is not JSON format because it missed the closing ",

I'm not sure if this is the main problem or just one of many problems that you encounter here ¯\_(ツ)_/¯

Maybe you should jump on the Splunk community Slack channel #getting-data-in and if your not yet on that slack signup here http://splk.it/slack

cheers, MuS

0 Karma
Reply

jayaprasad800
Engager
‎01-06-2020 07:29 PM

Hi Mus,

The logs are in json format only, I have used JQ to format it.

Also I have tried forwarding both logs( the plain json file and the formatted file with jq), but still same issue. splunk is not showing the full log file.

Sure, I will check in the slack channel as well.

thank you.

0 Karma
Reply
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...