topic Re: Can not choose default sourcetype=_json while onboarding in Getting Data In

Can not choose default sourcetype=_json while onboarding

Sandivsu — Tue, 27 Feb 2024 18:24:06 GMT

We can not choose default source type _json while onboarding.

Need to extract the json data within the log file, which is essential for an app owner.

log format - 2024-01-01T09:50:44+01:00 hostname APP2SAP[354]: {JSON data}

I have a splunk intermediate forwarder read these log files. Log file has non-json data followed by json data which bread n butter for application team (log format as shown above).

If I forward the data as-is to splunk, extraction is not proper, since it has non-json data at beginning.

Now, I need props n (or) transforms to extract, which I am not sure how.

Re: Can not choose default sourcetype=_json while onboarding

VatsalJagani — Wed, 28 Feb 2024 05:21:14 GMT

@Sandivsu - Not sure if you can do that with props and transforms. But I'll provide a solution you can apply at the search query level.

index=<your-index> ..... | rex field=_raw "\s\w+\[\w+\]:\s(?<json_content>\{.*\})" | spath input=json_content

I hope this helps!!! Kindly upvote if it does!!!

Re: Can not choose default sourcetype=_json while onboarding

PickleRick — Wed, 28 Feb 2024 11:53:06 GMT

Unfortunately, at the moment Splunk cannot automatically extract the structured data if it's not the whole event (as in your case - the json part is preceeded by non-json header).

There is an open idea for that https://ideas.splunk.com/ideas/EID-I-208

So far you can either parse the json part in search with help of the spath command as @VatsalJagani already showed or cut away the header part using SEDCMD or INGEST_EVAL (possibly extracting indexed fields if needed prior to removing the non-structured part).

As a side note - you should _not_ use the _json sourcetype. Define your own