Splunk Search
Highlighted

How can I extract all fields from a log file including json definiton?

Path Finder

I have a log file which entries/lines look like this:

12:17:35.4641 Info {"message":"TestKevin execution ended","level":"Information","logType":"Default","timeStamp":"2018-09-19T12:17:35.4641435+00:00","fingerprint":"2ee56795-e30b-4c98-b6cc-166249d18375","windowsIdentity":"RPA1\\Robotics","machineName":"RPA1","processName":"TestKevin","processVersion":"1.0.6828.23354","fileName":"Main_Alf_Production","jobId":"d415160d-8bfd-4374-b0c4-b03a35316b79","robotName":"ROBOTICS","totalExecutionTimeInSeconds":34,"totalExecutionTime":"00:00:34"}

First of all, I want to extract the time(12:17:35.4641), the Status (Info), and all of the key-value pairs included in the json definition. It would be great to do this at indexing time, so I don't have to include this logic within my search. Any Ideas?

0 Karma
Highlighted

Re: How can I extract all fields from a log file including json definiton?

Path Finder

If the data is all JSON, then you need this in your props.conf: KV_MODE=json

http://docs.splunk.com/Documentation/Splunk/7.1.3/Admin/Propsconf

0 Karma
Highlighted

Re: How can I extract all fields from a log file including json definiton?

Path Finder

I guess that's exactly my problem, that not all of my data is JSON. The first two fields are not part of the JSON definition and therefore this option does not seem to have any impact on the indexed data.

My workaround is to write the JSON to a single field (so I have 3 fields: "Time", "Status", "JSON definition") and the use the "spath" command to expand the Key-Value pairs.

index=<someindex> | spath input=jsonData 

jsonData is the field with the complete JSON definition.

But it would be nice to do this at indexing time. Any other suggestions?

0 Karma
Highlighted

Re: How can I extract all fields from a log file including json definiton?

Path Finder

So is your example the raw data?

0 Karma
Highlighted

Re: How can I extract all fields from a log file including json definiton?

Path Finder

Exactly, the example data above would be the _raw data.

But what I could do is to split up the fields "Time" "Status" and the JSON definition into 3 different fields (as explained in the comment to your first answer), if that would help. So we would just need to split up the key-value pairs of the field with the JSON definition.

0 Karma