topic Re: oneshot json | kvmode is incorrect in Getting Data In

Help with oneshot json: Why is | kvmode incorrect?

NullZero — Wed, 19 Oct 2022 16:22:40 GMT

I'm using a distributed Splunk Enterprise environment with over 15 peers at the Indexer Tier. I have some JSON data in a small file less than 500KB and I'm confident that the JSON is parsed correctly and this has been verified in Python with a simple check script.

issued command:

./splunk add oneshot "/tmp/<file.json>" -sourcetype xxxx:xxxx -index <index>

The command completes and the data is ingested.

However, it has parsed as an event per line and not as JSON. Obviously in props.conf the default is not set for 'KV_MODE = json'. There is no option in the CLI when using oneshot to set as JSON.

Any thoughts or guidance please. I am a certified Splunk PS consultant but everyday brings something new for all of us right.

Re: oneshot json | kvmode is incorrect

richgalloway — Mon, 17 Oct 2022 16:55:14 GMT

The sourcetype specified in the oneshot command should be one that properly processes JSON.

Re: oneshot json | kvmode is incorrect

NullZero — Mon, 17 Oct 2022 19:36:19 GMT

Thanks @richgalloway that seems like a really obvious solution now you say it. Referencing the docs I think therefore I should trial:

_json
json_no_timestamp

I appreciate the feedback and I will let you and the community know. The obvious drawback here is that you can't use a custom sourcetype per the client environment but I suppose oneshot is not designed for scale and batch or monitor should be used for the sustainable solution.

Re: oneshot json | kvmode is incorrect

NullZero — Tue, 18 Oct 2022 14:03:55 GMT

The following did ingest the data as JSON, and provide KV pairs:

./splunk add oneshot "/tmp/<file.json>" -sourcetype _json -index <index>

Re: Help with oneshot json: Why is | kvmode incorrect?

anwarmian — Sat, 07 Sep 2024 16:27:21 GMT

We need to keep in mind that KV_MODE applies to search time only and the field extractons ae best to be done at search time. Therefore, at index time if you have the following parameters set you should be good.

props.conf

[sourcetypename]
LINE_BREAKER
TIME_PREFIX
MAX_TIMESTAMP_LOOKAHEAD
TIME_FORMAT
TRUNCATE
SHOULD_LINEMERGE = false # LINE_BREAKER should be properly set so you can keep SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true