How to Parse raw events with some json data to JSON format

[devsru]

Hi All,

I am trying to parse raw data with json elements to proper JSON format in Splunk. I have tried multiple props.conf but failed to parse it as per expected output. Below I have attached the data coming as a single event on Splunk and expected data what we want to see. Can someone please correct my props.conf ?

Events on Splunk with default sourcetype

{"messageType":"DATA_MESSAGE","owner":"381491847064","logGroup":"tableau-cluster","logStream":"SentinelOne Agent Logs","subscriptionFilters":["splunk"],"logEvents":[{"id":"38791169637844522680841662226148491272212438883591651328","timestamp":1739456206172,"message":"[2025-02-13 15:16:41.413885] [110775] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24388.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24388.tmp: No such file or directory\n[2025-02-13 15:16:42.213970] [110823] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/hyper_transient.112335.24390.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/hyper_transient.112335.24390.tmp: No such file or directory\n[2025-02-13 15:16:42.214870] [110830] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24389.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24389.tmp: No such file or directory\n[2025-02-13 15:16:42.218488] [110823] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24391.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24391.tmp: No such file or directory\n[2025-02-13 15:16:43.815051] [110827] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24392.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24392.tmp: No such file or directory\n[2025-02-13 15:16:44.617525] [110773] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24394.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24394.tmp: No such file or directory\n[2025-02-13 15:16:45.413954] [110823] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24393.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24393.tmp: No such file or directory"},{"id":"38791169749325947928296247310685546917181598051987750913","timestamp":1739456211171,"message":"[2025-02-13 15:16:47.014642] [110770] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/hyper_transient.112335.24395.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/hyper_transient.112335.24395.tmp: No such file or directory\n[2025-02-13 15:16:47.813934] [110823] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24396.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24396.tmp: No such file or directory\n[2025-02-13 15:16:47.814459] [110828] [warning] DV process create: Couldn't fetch grandparent process of process 26395 from the data model\n[2025-02-13 15:16:47.815399] [110828] [warning] DV process create: Couldn't fetch grandparent process of process 26396 from the data model\n[2025-02-13 15:16:47.816855] [110827] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24397.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24397.tmp: No such file or directory\n[2025-02-13 15:16:48.616944] [110825] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream

Expected Output with fiedls extraction

{
    "messageType": "DATA_MESSAGE",
    "owner": "381491847064",
    "logGroup": "tableau-cluster",
    "logStream": "SentinelOne Agent Logs",
    "subscriptionFilters": ["splunk"],
    "logEvents": [
        {
            "id": "38791169637844522680841662226148491272212438883591651328",
            "timestamp": 1739456206172,
            "message": "[2025-02-13 15:16:41.413885] [110775] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24388.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24388.tmp: No such file or directory\n[2025-02-13 15:16:42.213970] [110823] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/hyper_transient.112335.24390.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/hyper_transient.112335.24390.tmp: No such file or directory\n[2025-02-13 15:16:42.214870] [110830] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24389.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24389.tmp: No such file or directory\n[2025-02-13 15:16:42.218488] [110823] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24391.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24391.tmp: No such file or directory\n[2025-02-13 15:16:43.815051] [110827] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24392.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24392.tmp: No such file or directory\n[2025-02-13 15:16:44.617525] [110773] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24394.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24394.tmp: No such file or directory\n[2025-02-13 15:16:45.413954] [110823] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24393.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24393.tmp: No such file or directory"
        }
    ]
}

Props.conf

[json_splunk_logs]
# Define the source type for the logs
sourcetype = json_splunk_logs

# Time configuration - Parse the timestamp in your message
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%6N
TIME_PREFIX = \["message"\] \[
# Specify how to break events in the multiline message
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)

# Event timestamp extraction
DATETIME_CONFIG = NONE

# JSON parsing - This tells Splunk to extract fields from JSON automatically
KV_MODE = json

# The timestamp is embedded in the message, so the following configuration is necessary for time extraction.
EXTRACT_TIMESTAMP = \["messageType":"DATA_MESSAGE","owner":"\d+","logGroup":"\w+","logStream":"\w+","subscriptionFilters":\[\\"splunk\\"\],\s"timestamp":(\d+),".*?

[richgalloway]

Unless there was an error copying into the post, the event does not parse because it is not well-formed JSON.

Also, the sourcetype attribute in props.conf is better placed in inputs.conf.  It's redundant in props.

[devsru]

Hi. Thanks for your reply but input is firehose AWS and we don’t have inputs. Is it possible for you to review my props.conf and if you can test in any dummy environment 

[richgalloway]

There is no need to test.  Splunk will only parse an event as JSON if the *entire* event is nothing but pure well-formed JSON.  It can't parse part of the event or extract a field and parse that.  Of course, you can do those things yourself in a query, but Splunk won't do it automatically.