Thank you @johnhuang  I tried as suggested but somehow the value for event_time is coming as blank in our use-case: Here is sample SPL which i'm using: index=cloud cloudaccount="account_id" sourcetype=lambda:project_name-sftp-logs-to-splunk "\"logGroup\": \"/aws/transfer/s-serverid\"" "CLOSE Path=\/projectname-bucketname-prd" "a.b.c.d.e.T0600420" | rex field=_raw max_match=0 "(?<Sent_At>\d+)[\r\n ,]+\"message\": \"[\w+.]+ CLOSE Path=\/projectname-bucketname-prd[-secure]*\/landing\/[prd\/]*(?<user_id>\w+)\/(?<FILE_NAME>[\w+.#]+) BytesIn=(?<Bytes_In>\d+)" | eval event_time=strftime((Sent_At/1000),"%m/%d/%Y %H:%M:%S") | table FILE_NAME user_id Bytes_In Sent_At event_time Result: FILE_NAME                 user_id             Bytes_In     Sent_At                           event_time ---------------------------------------------------------------------------------------------- a.b.c.d.e.T0600420 user_name 1   0                    1641294227380    Sample log events: 2022-01-04T11:04:02.452Z 9e7cf6d7-006a-4aea-8053-2aae3068f5b0 INFO Decoded payload: { "messageType": "DATA_MESSAGE", "owner": "account_id", "logGroup": "/aws/transfer/s-serverid", "logStream": "eft_nonprod.7eba87e510990497", "subscriptionFilters": [ "onepi-sftp-logs-to-splunk-filter" ], "logEvents": [ { "id": "36602084357565361730142322725669422549468033150221549568", "timestamp": 1641294227243, "message": "message-1" }, { "id": "36602084360419857115554242487785994488367023422987042817", "timestamp": 1641294227371, "message": "message-2" }, .... ] } ... View more

AIRFLOW_CTX_DAG_OWNER is printed like this in splunk logs: 2021-12-13T20:18:02.837Z 2b98fffe-7c1b-4dff-aedf-101c04bcd17d INFO Decoded payload: { "messageType": "DATA_MESSAGE", "owner": "765460745490", "logGroup": "airflow-OnePIAirflowEnvironment-DEV-Task", "logStream": "any_bash_command_dag/bash_command/2021-12-13T20_18_00.187744+00_00/1.log", "subscriptionFilters": [ "airflow_2_splunk_task" ], "logEvents": [ { "id": "36560436722252801980494271713377497999438999533834272768", "timestamp": 1639426682686, "message": "[2021-12-13 20:18:02,686] {{taskinstance.py:1283}} INFO - Exporting the following env vars:\nAIRFLOW_CTX_DAG_OWNER=airflow\nAIRFLOW_CTX_DAG_ID=any_bash_command_dag\nAIRFLOW_CTX_TASK_ID=bash_command\nAIRFLOW_CTX_EXECUTION_DATE=2021-12-13T20:18:00.187744+00:00\nAIRFLOW_CTX_DAG_RUN_ID=manual__2021-12-13T20:18:00.187744+00:00" } ] } The SPL without \n doesn't yield any result. index=cloud sourcetype=lambda:Airflow2Splunk "\"logGroup\"" "\"airflow-OnePIAirflowEnvironment-DEV-Task\"" | transaction startswith="AIRFLOW_CTX_DAG_OWNER" endswith="Marking task as FAILED". .... .... .... ... View more

When I run following SPL without \n it doesn't yield any result.  | transaction startswith="AIRFLOW_CTX_DAG_OWNER" endswith="Marking task as FAILED"........ However : | transaction startswith="\\nAIRFLOW_CTX_DAG_OWNER" endswith="Marking task as FAILED"........ or | transaction startswith="\nAIRFLOW_CTX_DAG_OWNER" endswith="Marking task as FAILED"........ both yields same results. '\n' is added in Splunk airflow logs (in JSON) when the airflow logs are being pushed to Splunk. That's why I added \\n before AIRFLOW_CTX_DAG_OWNER :   Airflow logs: [2021-12-13 20:20:13,145] {{logging_mixin.py:104}} INFO - Running <TaskInstance: any_bash_command_dag.bash_command 2021-12-13T20:20:10.033766+00:00 [running]> on host ip-10-223-50-200.ec2.internal [2021-12-13 20:20:13,567] {{taskinstance.py:1283}} INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=any_bash_command_dag AIRFLOW_CTX_TASK_ID=bash_command AIRFLOW_CTX_EXECUTION_DATE=2021-12-13T20:20:10.033766+00:00 AIRFLOW_CTX_DAG_RUN_ID=manual__2021-12-13T20:20:10.033766+00:00 Airflow logs in Splunk: 2021-12-13T20:18:02.837Z 2b98fffe-7c1b-4dff-aedf-101c04bcd17d INFO Decoded payload: { "messageType": "DATA_MESSAGE", "owner": "765460745490", "logGroup": "airflow-OnePIAirflowEnvironment-DEV-Task", "logStream": "any_bash_command_dag/bash_command/2021-12-13T20_18_00.187744+00_00/1.log", "subscriptionFilters": [ "airflow_2_splunk_task" ], "logEvents": [ { "id": "36560436722252801980494271713377497999438999533834272768", "timestamp": 1639426682686, "message": "[2021-12-13 20:18:02,686] {{taskinstance.py:1283}} INFO - Exporting the following env vars:\nAIRFLOW_CTX_DAG_OWNER=airflow\nAIRFLOW_CTX_DAG_ID=any_bash_command_dag\nAIRFLOW_CTX_TASK_ID=bash_command\nAIRFLOW_CTX_EXECUTION_DATE=2021-12-13T20:18:00.187744+00:00\nAIRFLOW_CTX_DAG_RUN_ID=manual__2021-12-13T20:18:00.187744+00:00" } ] } ... View more

Thanks again @ITWhisperer  I'm  seeing an issue during additional testing. Can you please review and let me know what wrong am I doing in SPL#2. SPL#1 : This lists dags, tasks_id, status (SUCCESS/FAILED), Rows_Affected, Task_Start_Date, Task_End_Date correctly: index=cloud sourcetype=lambda:Airflow2Splunk "\"logGroup\"" "\"airflow-OnePIAirflowEnvironment-DEV-Task\"" | transaction startswith="\\nAIRFLOW_CTX_DAG_OWNER" endswith="Marking task as" | rex field=_raw "AIRFLOW_CTX_DAG_OWNER"=(?<dag_owner>\w+) | rex field=_raw "AIRFLOW_CTX_DAG_ID=(?<dag_id>\w+)" | rex field=_raw "AIRFLOW_CTX_TASK_ID=(?<task_id>\w+)" | rex field=_raw max_match=0 "Rows affected: (?<rows_affected>\d+)" | eval rows_affected=mvjoin(rows_affected, ",") | rex field=_raw "Marking task as (?<status>[^\.]+)" | rex field=_raw "start_date=(?<task_start_date>\d{8}T\d{6})" | rex field=_raw "end_date=(?<task_end_date>\d{8}T\d{6})" | eval new_start_task_date=strptime(task_start_date,"%Y%m%dT%H%M%S") | eval start_task_date=strftime(new_start_task_date,"%Y-%m-%d %H:%M:%S") | eval new_end_task_date=strptime(task_end_date,"%Y%m%dT%H%M%S") | eval end_task_date=strftime(new_end_task_date,"%Y-%m-%d %H:%M:%S") | table dag_owner dag_id task_id status rows_affected start_task_date end_task_date SPL#2 This should list dags, tasks_id, status (FAILED), Rows_Affected, Task_Start_Date, Task_End_Date correctly: But this SPL is not working as expected. I can some tasks whose status is SUCCESS also in the list.  index=cloud sourcetype=lambda:Airflow2Splunk "\"logGroup\"" "\"airflow-OnePIAirflowEnvironment-DEV-Task\"" | transaction startswith="\\nAIRFLOW_CTX_DAG_OWNER" endswith="Marking task as FAILED" | rex field=_raw "AIRFLOW_CTX_DAG_OWNER"=(?<dag_owner>\w+) | rex field=_raw "AIRFLOW_CTX_DAG_ID=(?<dag_id>\w+)" | rex field=_raw "AIRFLOW_CTX_TASK_ID=(?<task_id>\w+)" | rex field=_raw max_match=0 "Rows affected: (?<rows_affected>\d+)" | eval rows_affected=mvjoin(rows_affected, ",") | rex field=_raw "Marking task as (?<status>[^\.]+)" | rex field=_raw "start_date=(?<task_start_date>\d{8}T\d{6})" | rex field=_raw "end_date=(?<task_end_date>\d{8}T\d{6})" | eval new_start_task_date=strptime(task_start_date,"%Y%m%dT%H%M%S") | eval start_task_date=strftime(new_start_task_date,"%Y-%m-%d %H:%M:%S") | eval new_end_task_date=strptime(task_end_date,"%Y%m%dT%H%M%S") | eval end_task_date=strftime(new_end_task_date,"%Y-%m-%d %H:%M:%S") | table dag_owner dag_id task_id status rows_affected start_task_date end_task_date ... View more

Thank you @ITWhisperer . your SPL is working as expected when I feed real data. I need to add two more columns in table : dag_run_date, dag_run_time, the values of those two column needs to be extracted from this statement but unable to form corresponding rex command . Can you help me on that? AIRFLOW_CTX_EXECUTION_DATE=2021-12-08T18:57:24.419709+00:00   ... View more

Thank you very much! @bowesmana  Appreciate your help! ... View more