Automatic JSON log extraction

deepak02 — Tue, 28 Mar 2017 17:35:31 GMT

Hi,

I am uploading logs in JSON format into Splunk.

I want to enable automatic field extraction.

Is there any setting for this, or does Splunk always enable automatic field extraction by default?

Thanks,
Deepak

Re: Automatic JSON log extraction

mattymo — Tue, 28 Mar 2017 17:49:21 GMT

Hi deepak02!

Splunk has both indexed extractions and searchtime extractions for json.

They are found in props.conf.
http://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Propsconf

INDEXED_EXTRACTIONS = < CSV|W3C|TSV|PSV|JSON >
* Tells Splunk the type of file and the extraction and/or parsing method
  Splunk should use on the file.
  CSV  - Comma separated value format
  TSV  - Tab-separated value format
  PSV  - pipe "|" separated value format
  W3C  - W3C Extended Extended Log File Format
  JSON - JavaScript Object Notation format
* These settings default the values of the remaining settings to the
  appropriate values for these known formats.
* Defaults to unset.

*If you are using a forwarder, be sure to put the props.conf on the forwarder! not just the indexer!

Also as an FYI, Splunk has a searchtime extractions available:

KV_MODE = [none|auto|auto_escaped|multi|json|xml]
* Used for search-time field extractions only.
* Specifies the field/value extraction mode for the data.
* Set KV_MODE to one of the following:
  * none: if you want no field/value extraction to take place.
  * auto: extracts field/value pairs separated by equal signs.
  * auto_escaped: extracts fields/value pairs separated by equal signs and
                  honors \" and \\ as escaped sequences within quoted
                  values, e.g field="value with \"nested\" quotes"
  * multi: invokes the multikv search command to expand a tabular event into
           multiple events.
  * xml : automatically extracts fields from XML data.
  * json: automatically extracts fields from JSON data.
* Setting to 'none' can ensure that one or more user-created regexes are not
  overridden by automatic field/value extraction for a particular host,
  source, or source type, and also increases search performance.
* Defaults to auto.
* The 'xml' and 'json' modes will not extract any fields when used on data
  that isn't of the correct format (JSON or XML).

AUTO_KV_JSON = [true|false]
* Used for search-time field extractions only.
* Specifies whether to try json extraction automatically.
* Defaults to true.

What ever way you decide, I encourage you to try a sample of your json using the Add Data wizard, to ensure you are getting the extractions you expect.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Setsourcetype

topic Re: Automatic JSON log extraction in Getting Data In

Automatic JSON log extraction

Re: Automatic JSON log extraction