topic Re: JSON file event breaking parsing on universal forwarder in Getting Data In

JSON file event breaking parsing on universal forwarder

rayar — Thu, 12 Mar 2020 13:09:06 GMT

I have a JSON file.
Once I upload the file on the search head using the below stanza in props.conf it's indexed properly.

Splunk 7.3.4

[json_test]
CHARSET = UTF-8
DATETIME_CONFIG = CURRENT
SEDCMD-cut_footer = s/\]\,\n\s*\"total\":.*$/g
SEDCMD-cut_header = s/^\{\n\s*\"matches\":\s\[/g
category = Structured
disabled = false
HEADER_FIELD_LINE_NUMBER = 3
SHOULD_LINEMERGE = 0
TRUNCATE = 0
INDEXED_EXTRACTIONS = json
KV_MODE = none

Once I upload the data from UF the data do not break to events

Universal Forwarder

props.conf

[json_test]
CHARSET = UTF-8
INDEXED_EXTRACTIONS = json

inputs.conf

[monitor:///tmp/*.json]
disabled = 0
sourcetype = json_test
index = test_hr
crcSalt  = REINDEXMEPLEASE
initCrcLength = 780

Indexer

props.conf

[json_test]
DATETIME_CONFIG = CURRENT
SEDCMD-cut_footer = s/\]\,\n\s*\"total\":.*$/g
SEDCMD-cut_header = s/^\{\n\s*\"matches\":\s\[/g
category = Structured
disabled = false
HEADER_FIELD_LINE_NUMBER = 3
SHOULD_LINEMERGE = 0
TRUNCATE = 0

Search Head

props.conf

[json_test]
KV_MODE = none

Re: JSON file event breaking parsing on universal forwarder

FrankVl — Fri, 13 Mar 2020 08:23:26 GMT

What if you remove the INDEXED_EXTRACTIONS = json from the UF's config (and enable kvmode again, or move the indexed extractions to the indexer)? The UF will try to do the json extractions, without any of the custom line breaking and header stripping. And once the indexed extractions have been done, the downstream splunk enterprise instance will no longer apply linebreaking stuff if I'm not mistaken.

Indexing JSON files that contain multiple events in one json structure is anyway a pain in the proverbial butt. You might also want to look at setting sensible EVENT_BREAKER settings on your UF, to at least make sure events arrive in one piece at the indexers.

Or consider using a heavy forwarder for this, so that indexed extractions and linebreaking and such happen at the same place.

But all in all, I think changing how this data gets logged, or do some pre-processing on the json file to transform it into individual events, might be the best (but not necessarily the easiest) thing to do here.

Re: JSON file event breaking parsing on universal forwarder

rayar — Wed, 30 Sep 2020 04:38:27 GMT

Hi
thanks for your inputs

updated as below

CHARSET = UTF-8
KV_MODE = none

Indexer

[json_odelia]
DATETIME_CONFIG = CURRENT
SEDCMD-cut_footer = s/]\,\n\s*\"total\":.$/g
SEDCMD-cut_header = s/^{\n\s\"matches\":\s[/g
category = Structured
disabled = false
HEADER_FIELD_LINE_NUMBER = 3
SHOULD_LINEMERGE = 0
TRUNCATE = 0
INDEXED_EXTRACTIONS = json

still not working