topic Re: Issue with consuming JSON output from shell script in Getting Data In

Issue with consuming JSON output from shell script

harishgopalan — Mon, 28 Sep 2020 14:28:11 GMT

Dear Splunk Dev,

This is a very fundamental question.

If I've a shell script that produces a JSON type of output such as {k1:v1,k2:v2,:k3:v3} can I consume it as JSON in Splunk when it is indexed ?

I tried to check this by doing the following in splunk version : Splunk 5.0.4 build 172409.

Created an app called test_demo
Created an index called test_demo
Created a script data input mapped to the shell script location : /opt/splunk/etc/apps/test_demo/bin/scripts/json_test.sh with output going to test_demo and source type as test_demo.
Created a props.conf at location : /opt/splunk/etc/apps/test_demo/local/ with the following content

[test_demo]
CHARSET = UTF-8
NO_BINARY_CHECK = 1
TIME_FORMAT = %a %b %d %H:%M:%S %z %Y
TIME_PREFIX = "__time":"
MAX_TIMESTAMP_LOOKAHEAD = 150
SHOULD_LINEMERGE = false
TZ = UTC
KV_MODE = json

Now if I run a search using the index i.e index=test_demo, it doesn't recognize the input as {k1:v1,k2:v2,:k3:v3} instead it shows as one single string.

Could you please help me with this issue ?

Regards
Harish

Re: Issue with consuming JSON output from shell script

harishgopalan — Wed, 31 Jul 2013 18:48:01 GMT

Can someone please help me with this issue ?

Re: Issue with consuming JSON output from shell script

Gilberto_Castil — Wed, 31 Jul 2013 19:37:13 GMT

Splunk will ingest JSON data, validate its syntax and display in a friendly manner. That is a confirmed assertion so I might to check on the syntax of your JSON string. For instance, consider this screenshot of JSON data indexed into Splunk.

Checking on the sample given, this syntax may not be a pure JSON string.

{
  k1:v1,
  k2:v2,
  :k3:v3
}

You can check your syntax JSONLint if you are unsure. In this case Splunk sees the data and shows it like a text string.

This would be the same content with the correct syntax. Notice how Splunk recognizes the JSON syntax and it also finds the fields, two alpha and one numeric.

  {
    "k1": "v1",
    "k2": "v2",
    "k3": 3
  }

Re: Issue with consuming JSON output from shell script

harishgopalan — Thu, 01 Aug 2013 02:23:30 GMT

Got it Gilberto. Thanks a lot.