Getting Data In

Why is Splunk not parsing JSON data correctly with my current sourcetype configuration?

rusty009
Path Finder

I am trying to import JSON objects into splunk, my sourcetype is below,

[ _json_cloudflare ]
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TIMESTAMP_FIELDS=timestamp
TIME_FORMAT=%s%9N
TIME_PREFIX=^
category=Structured
description=JSON cloudflare
disabled=false
pulldown_type=true

This works flawlessly on the search head, but when I index the data, it goes into Splunk like below,

alt text

The sourcetype is on both the universal forwarder & the indexer, any ideas what is wrong?

0 Karma
1 Solution

Yorokobi
SplunkTrust
SplunkTrust

JSON is a wonderful data structure that Splunk handles beautifully so long as it is 100% JSON and Splunk is properly informed about it.

So, a few things I would do:

  1. Don't use a _ to prefix your sourcetype names. I know, Splunk does that in the UI but I'm of the opinion _names should only be used for internal-to-Splunk data types, field names, etc.
  2. Remove the spaces inside the stanza declaration: [good_stanza] vs. [ bad_stanza ]
  3. Place only the relevant bits of the props.conf where they matter.

    props.conf on the universal forwarder and search head:

    [json_sourcetype]
    INDEXED_EXTRACTIONS = JSON

    props.conf on the indexer(s):

    [json_sourcetype]
    TIME_PREFIX = timestamp:
    TIME_FORMAT = %s%3N
    MAX_TIMESTAMP_LOOKAHEAD = 15

I removed TIMESTAMP_FIELDS in favour of TIME_PREFIX so that time extraction will work in the event field extractions fail.

View solution in original post

cyndiback
Path Finder

How are you bringing in CloudFlare logs?

0 Karma

Yorokobi
SplunkTrust
SplunkTrust

JSON is a wonderful data structure that Splunk handles beautifully so long as it is 100% JSON and Splunk is properly informed about it.

So, a few things I would do:

  1. Don't use a _ to prefix your sourcetype names. I know, Splunk does that in the UI but I'm of the opinion _names should only be used for internal-to-Splunk data types, field names, etc.
  2. Remove the spaces inside the stanza declaration: [good_stanza] vs. [ bad_stanza ]
  3. Place only the relevant bits of the props.conf where they matter.

    props.conf on the universal forwarder and search head:

    [json_sourcetype]
    INDEXED_EXTRACTIONS = JSON

    props.conf on the indexer(s):

    [json_sourcetype]
    TIME_PREFIX = timestamp:
    TIME_FORMAT = %s%3N
    MAX_TIMESTAMP_LOOKAHEAD = 15

I removed TIMESTAMP_FIELDS in favour of TIME_PREFIX so that time extraction will work in the event field extractions fail.

View solution in original post

rusty009
Path Finder

Thanks, that worked !

0 Karma

somesoni2
Revered Legend

I don't see any issues here. The JSON parser of Splunk Web shows the JSON syntax highlighted, and that means the indexed data is correctly parsed as JSON.

If you want to see the actual raw data without highlighting, click on the "Show as raw text" hyperlink below the event.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!