topic Re: Collect command to index in JSON format in Getting Data In

Collect command to index in JSON format

alvaromari83 — Thu, 23 May 2019 06:31:43 GMT

Hello!

I'm trying to generate some summarized data by using the collect command in my SPL queries.

The event format by default is

05/20/2019 07:15:00 +0200, info_min_time=1.000, info_max_time=1558557310.000, info_search_time=1558557310.410, field1=value1 field2=value2

Some of this fields, are actually JSON objects, like:

05/20/2019 07:15:00 +0200, info_min_time=1.000, info_max_time=1558557310.000, info_search_time=1558557310.410, field1=value1 field2=value2 field3="{"subfield1":"subvalue1", "subfield2":"subvalue2",...}"

So, to avois a mixture of formats in my collected event, I would like to index the results, using the collect command, BUT in JSON format for the whole event, something like:

{"time":"05/20/2019 07:15:00 +0200", "info_min_time"=1.000, "info_max_time":1558557310.000, "info_search_time":1558557310.410, results: {"field1":"value1" "field2":"value2", field3="{"subfield1":"subvalue1", "subfield2":"subvalue2",...}"...}}

Is such thing feasible? Some props tweaking?

Thank you in advance!

Re: Collect command to index in JSON format

koshyk — Thu, 23 May 2019 07:25:55 GMT

You can look into it from two perspective
1. Is the source data the most important thing in your organisation?
2. If the data format is important

In many organisation (1) is more important. This is to preserve the original data. Splunk doesn't have any issue in extracting data even if the JSON format is mixed. You index "original" data, but just mention the json part to extract automatically

Please see this thread on how to index json mixed format. https://answers.splunk.com/answers/117121/extract-json-data-within-the-logs-json-mixed-with.html

You could even add a stanza just to extract pure JSON if need be at search time. This way you preserve the original data as well as you get key-value at search-time

Re: Collect command to index in JSON format

kamlesh_vaghela — Thu, 23 May 2019 07:35:22 GMT

@alvaromari83

Can you please try | mkjson command from JSON Tools Splunk App ?

https://splunkbase.splunk.com/app/3540/#/overview

Re: Collect command to index in JSON format

kamlesh_vaghela — Thu, 23 May 2019 07:35:44 GMT

https://github.com/doksu/TA-jsontools/wiki