How to Parse raw events with some json data to JSON format

devsru — Thu, 13 Feb 2025 18:36:39 GMT

Hi All,

I am trying to parse raw data with json elements to proper JSON format in Splunk. I have tried multiple props.conf but failed to parse it as per expected output. Below I have attached the data coming as a single event on Splunk and expected data what we want to see. Can someone please correct my props.conf ?

Events on Splunk with default sourcetype

{"messageType":"DATA_MESSAGE","owner":"381491847064","logGroup":"tableau-cluster","logStream":"SentinelOne Agent Logs","subscriptionFilters":["splunk"],"logEvents":[{"id":"38791169637844522680841662226148491272212438883591651328","timestamp":1739456206172,"message":"[2025-02-13 15:16:41.413885] [110775] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24388.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24388.tmp: No such file or directory\n[2025-02-13 15:16:42.213970] [110823] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/hyper_transient.112335.24390.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/hyper_transient.112335.24390.tmp: No such file or directory\n[2025-02-13 15:16:42.214870] [110830] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24389.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24389.tmp: No such file or directory\n[2025-02-13 15:16:42.218488] [110823] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24391.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24391.tmp: No such file or directory\n[2025-02-13 15:16:43.815051] [110827] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24392.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24392.tmp: No such file or directory\n[2025-02-13 15:16:44.617525] [110773] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24394.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24394.tmp: No such file or directory\n[2025-02-13 15:16:45.413954] [110823] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24393.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24393.tmp: No such file or directory"},{"id":"38791169749325947928296247310685546917181598051987750913","timestamp":1739456211171,"message":"[2025-02-13 15:16:47.014642] [110770] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/hyper_transient.112335.24395.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/hyper_transient.112335.24395.tmp: No such file or directory\n[2025-02-13 15:16:47.813934] [110823] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24396.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24396.tmp: No such file or directory\n[2025-02-13 15:16:47.814459] [110828] [warning] DV process create: Couldn't fetch grandparent process of process 26395 from the data model\n[2025-02-13 15:16:47.815399] [110828] [warning] DV process create: Couldn't fetch grandparent process of process 26396 from the data model\n[2025-02-13 15:16:47.816855] [110827] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24397.tmp: stat failed on path: /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream.112335.24397.tmp: No such file or directory\n[2025-02-13 15:16:48.616944] [110825] [error] full_file_overwrite_flag: failed to stat /app/tableau/tableau_data/data/tabsvc/temp/hyper_0.20233.24.0718.1630/copyexternalstream

Expected Output with fiedls extraction

Props.conf

[json_splunk_logs] # Define the source type for the logs sourcetype = json_splunk_logs # Time configuration - Parse the timestamp in your message TIME_FORMAT = %Y-%m-%d %H:%M:%S.%6N TIME_PREFIX = \["message"\] \[ # Specify how to break events in the multiline message SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) # Event timestamp extraction DATETIME_CONFIG = NONE # JSON parsing - This tells Splunk to extract fields from JSON automatically KV_MODE = json # The timestamp is embedded in the message, so the following configuration is necessary for time extraction. EXTRACT_TIMESTAMP = \["messageType":"DATA_MESSAGE","owner":"\d+","logGroup":"\w+","logStream":"\w+","subscriptionFilters":\[\\"splunk\\"\],\s"timestamp":(\d+),".*?

Re: How to Parse raw events with some json data to JSON format

richgalloway — Thu, 13 Feb 2025 19:10:52 GMT

Unless there was an error copying into the post, the event does not parse because it is not well-formed JSON.

Also, the sourcetype attribute in props.conf is better placed in inputs.conf. It's redundant in props.

Re: How to Parse raw events with some json data to JSON format

devsru — Thu, 13 Feb 2025 19:32:33 GMT

Hi. Thanks for your reply but input is firehose AWS and we don’t have inputs. Is it possible for you to review my props.conf and if you can test in any dummy environment

Re: How to Parse raw events with some json data to JSON format

richgalloway — Fri, 14 Feb 2025 00:36:57 GMT

There is no need to test. Splunk will only parse an event as JSON if the *entire* event is nothing but pure well-formed JSON. It can't parse part of the event or extract a field and parse that. Of course, you can do those things yourself in a query, but Splunk won't do it automatically.

topic Re: How to Parse raw events with some json data to JSON format in Getting Data In

How to Parse raw events with some json data to JSON format

Re: How to Parse raw events with some json data to JSON format

Re: How to Parse raw events with some json data to JSON format

Re: How to Parse raw events with some json data to JSON format