How to parse/format json at search-time?

[dbanerjee17]

New in handling json files in splunk with little knowledge in rex. Need help on parsing/formatting json logs in search time. Recently collected auth0 json logs. Every event starts with different character and ends with different character. Looks like Splunk break based on the max limit of the characters per event. I am trying to read the data in Search with proper formatting in table form with all fields.

props.conf stanza:

[auth0]
INDEXED_EXTRACTIONS = json
KV_MODE = none

logs looks like:

ol: ...: {}\n...., ...."}, hostname":"...","user_id":"...", user_name":"...", "strategy":"...", "strategy_type":"enterprise","_id":"...",  "log_id":"...",
"isMobile":...},{"date":"2018-03-21T23:28:59.296Z","type":"...","connection_id":"","client_id":"...","client_name":"...", "ip":"...","user_agent":"...", 
"details":{"return_to":"...","allowed_logout_url":[..."]},"hostname":"...","user_id":"...","user_name":"...","_id":"...","log_id":"...","isMobile":...},
{"date":"2018-03-21T23:28:59.286Z","type":"...","connection":"...","connection_id":"...","client_id":"...","client_name":"...","ip":"...","user_agent":"...",
"details":{"prompts":[{"name":"....","initiatedAt":...,
....several line from json file

Sometimes the event starts with [{"date", sometimes with something else and ends before closing }.
I went through the logs and understood that, it should be something like below in each event.

{   "date":"2018-03-09T23:41:05.437Z",
        "type":"..",
        "connection":"...
        "connection_id":"...",
        "client_id":"...",
        "client_name":"...",
        "ip":"...",
        "user_agent":"...",
        "details":{
            "timings":{
                "connection":...,
                "all_rules":...,
                "internal":...,
                "total":...
                    },
            "stats":{
                "loginsCount":...
                    },
            "consoleOut":"..."
                    },
        "hostname":"...",
        "user_id":"...",
        "user_name":"...",
        "strategy":"...",
        "strategy_type":"...",
        "_id":"...",
        "log_id":"...",
        "isMobile":...
    },

ends up comma and then another one starts. Is there any way, I can extract log in search. Please advise.

[venkatasri]

Hi @akj 

If your json well structured then event_breaking works fine hence you can see the fields extracted correctly on search UI.

#props.conf should be deployed to UF along with additional setting you want.

[sourcetype]

INDEXED_EXTRACTIONS = json

----

in Search UI did you try spath? Works only if json is well structured.

index=<idx> sourcetype=<your_st> | spath 

---

An upvote would be appreciated if this reply helps!

[akj]

Hi, facing same issue , did you found a solution for it.