topic Ingesting JSON formatted logs into Splunk in Getting Data In

Ingesting JSON formatted logs into Splunk

lball — Wed, 27 Feb 2019 22:17:02 GMT

I'm able to get JSON formatted linux os & modx web logs into a Splunk index, but they are not formatted or parsed. How can I get the logs to be efficiently parsed into the index so that they can be searched and used for reporting & dashboards. If this is impractical, is there a better way to get modx web logs into Splunk? If I am able to get them sent in syslog format will they parse correctly?

Re: Ingesting JSON formatted logs into Splunk

richgalloway — Wed, 27 Feb 2019 22:22:14 GMT

What are the props.conf settings for that sourcetype?

Re: Ingesting JSON formatted logs into Splunk

jeffbat — Wed, 27 Feb 2019 22:31:59 GMT

If you can grab a copy of the file you are trying to read, then on a dev splunk instance walk through the Add Data function in the web console.

Just import your file directly and when at the Set Source Type, choose, Structured->_json

You can then make sure it looks like it is parsing correctly and do a Save As to a new name/sourcetype name. Then when you finish getting it all read in, you can go to your drive and look for the inputs/props/transforms conf files it would create. Then you can use those on the forwarder you are trying to read the file originally from (or pushed out through a deployment server in an app).

Re: Ingesting JSON formatted logs into Splunk

worshamn — Thu, 28 Feb 2019 04:19:14 GMT

Like richgalloway mentioned in props.conf, make sure it has set KV_MODE = json. Also make sure that each event is a complete JSON event (for example doesn't have any text written before the JSON)

You could always copy a JSON line and paste it into a JSON pretty print web site to make sure they can parse it, like https://jsonformatter.org/json-pretty-print.

Re: Ingesting JSON formatted logs into Splunk

hookupgeek — Mon, 02 Sep 2019 13:45:36 GMT

Thanks for the tip!