Getting Data In

Ingesting JSON formatted logs into Splunk

lball
Explorer

I'm able to get JSON formatted linux os & modx web logs into a Splunk index, but they are not formatted or parsed. How can I get the logs to be efficiently parsed into the index so that they can be searched and used for reporting & dashboards. If this is impractical, is there a better way to get modx web logs into Splunk? If I am able to get them sent in syslog format will they parse correctly?

Tags (2)
0 Karma

jeffbat
Path Finder

If you can grab a copy of the file you are trying to read, then on a dev splunk instance walk through the Add Data function in the web console.

Just import your file directly and when at the Set Source Type, choose, Structured->_json

You can then make sure it looks like it is parsing correctly and do a Save As to a new name/sourcetype name. Then when you finish getting it all read in, you can go to your drive and look for the inputs/props/transforms conf files it would create. Then you can use those on the forwarder you are trying to read the file originally from (or pushed out through a deployment server in an app).

0 Karma

hookupgeek
New Member

Thanks for the tip!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

What are the props.conf settings for that sourcetype?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

worshamn
Contributor

Like richgalloway mentioned in props.conf, make sure it has set KV_MODE = json. Also make sure that each event is a complete JSON event (for example doesn't have any text written before the JSON)

You could always copy a JSON line and paste it into a JSON pretty print web site to make sure they can parse it, like https://jsonformatter.org/json-pretty-print.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.