How to ping Federate cloud logs on splunk?

[vikashjha]

Hi,

We have onboarded ping federate logs in splunk but we are getting multiple logs getting clubbed in one. Can someone help me why is this happening and how can I rectify this. Below is the log sample.

{"owner": "689186784177", "logGroup": "/aws/containerinsights/prod/application", "logStream": "pingaccess-was-admin-0_ping-cloud_pingaccess-was-admin", "logEvents": [{"id": "37013848036576111097453574573076270650086865051558412288", "timestamp": 1659758349197, "message": {"log": "<134>Aug 6 03:59:04 pingaccess-was-admin-0 , {\"thread\":\"ReqProc-StageThread[2]/LAUQHf9OalMaVcCzi173zw\",\"level\":\"INFO\",\"loggerName\":\"apiaudit\",\"message\":\"PA Audit with data \",\"endOfBatch\":false,\"loggerFqcn\":\"org.apache.logging.slf4j.Log4jLogger\",\"instant\":{\"epochSecond\":1659758344,\"nanoOfSecond\":121700000},\"threadId\":64,\"threadPriority\":5,\"date\":\"2022-08-06T03:59:04+0000\",\"exchangeId\":\"LAUQHf9OalMaVcCzi173zw\",\"roundTripMS\":\"1\",\"subject\":\"Administrator\",\"authMech\":\"Basic\",\"client\":\"127.0.0.1\",\"method\":\"GET\",\"resource\":\"* [] /pa-admin-api/v3 /version:-1\",\"requestUri\":\"/pa-admin-api/v3/version\",\"responseCode\":\"200\",\"logPurpose\":\"Audit_SIEM\"}\r\n", "stream": "stdout", "docker": {"container_id": "074987ed295dde1388b9e983bfe7bef9e7140a420a16070a42b57cc5a68ba0be"}, "kubernetes": {"container_name": "pingaccess-was-admin", "namespace_name": "ping-cloud", "pod_name": "pingaccess-was-admin-0", "container_image": "public.ecr.aws/r2h3l6e4/pingcloud-apps/pingaccess-was:6.3.1-v1.0.19", "container_image_id": "docker-pullable://public.ecr.aws/r2h3l6e4/pingcloud-apps/pingaccess-was@sha256:f5d3fea0441c96815e1c89d108d9e57c3a8db0e3809a7c597a5000754a9b22ec", "pod_id": "1b0c2182-cdf3-44aa-aecb-26d9ed968169", "host": "ip-10-10-117-227.us-east-2.compute.internal", "labels": {"app": "ping-cloud", "cluster": "pingaccess-was-cluster", "controller-revision-hash": "pingaccess-was-admin-66c966594f", "role": "pingaccess-was-admin", "statefulset_kubernetes_io/pod-name": "pingaccess-was-admin-0"}, "master_url": "https://172.20.0.1:443/api", "namespace_id": "38da143e-d00d-489a-9284-560673491f1d", "namespace_labels": {"app": "ping-cloud", "app_kubernetes_io/instance": "ping-cloud-master-us-east-2", "kubernetes_io/metadata_name": "ping-cloud"}}}}, {"id": "37013848148102137835305220903896397731601320942966472705", "timestamp": 1659758354198, "message": {"log": "<134>Aug 6 03:59:09 pingaccess-was-admin-0 , {\"thread\":\"ReqProc-StageThread[4]/vbAQ-x_FAX3vzYPGAFJILA\",\"level\":\"INFO\",\"loggerName\":\"apiaudit\",\"message\":\"PA Audit with data \",\"endOfBatch\":false,\"loggerFqcn\":\"org.apache.logging.slf4j.Log4jLogger\",\"instant\":{\"epochSecond\":1659758349,\"nanoOfSecond\":145851000},\"threadId\":69,\"threadPriority\":5,\"date\":\"2022-08-06T03:59:09+0000\",\"exchangeId\":\"vbAQ-x_FAX3vzYPGAFJILA\",\"roundTripMS\":\"1\",\"subject\":\"Administrator\",\"authMech\":\"Basic\",\"client\":\"127.0.0.1\",\"method\":\"GET\",\"resource\":\"* [] /pa-admin-api/v3 /version:-1\",\"requestUri\":\"/pa-admin-api/v3/version\",\"responseCode\":\"200\",\"logPurpose\":\"Audit_SIEM\"}\r\n", "stream": "stdout", "docker": {"container_id": "074987ed295dde1388b9e983bfe7bef9e7140a420a16070a42b57cc5a68ba0be"}, "kubernetes": {"container_name": "pingaccess-was-admin", "namespace_name": "ping-cloud", "pod_name": "pingaccess-was-admin-0", "container_image": "public.ecr.aws/r2h3l6e4/pingcloud-apps/pingaccess-was:6.3.1-v1.0.19", "container_image_id": "docker-pullable://public.ecr.aws/r2h3l6e4/pingcloud-apps/pingaccess-was@sha256:f5d3fea0441c96815e1c89d108d9e57c3a8db0e3809a7c597a5000754a9b22ec", "pod_id": "1b0c2182-cdf3-44aa-aecb-26d9ed968169", "host": "ip-10-10-117-227.us-east-2.compute.internal", "labels": {"app": "ping-cloud", "cluster": "pingaccess-was-cluster", "controller-revision-hash": "pingaccess-was-admin-66c966594f", "role": "pingaccess-was-admin", "statefulset_kubernetes_io/pod-name": "pingaccess-was-admin-0"}, "master_url": "https://172.20.0.1:443/api", "namespace_id": "38da143e-d00d-489a-9284-560673491f1d", "namespace_labels": {"app": "ping-cloud", "app_kubernetes_io/instance": "ping-cloud-master-us-east-2", "kubernetes_io/metadata_name": "ping-cloud"}}}}], "envType": "prod"}

{"owner": "689186784177", "logGroup": "/aws/containerinsights/prod/application", "logStream": "server_logs.pingfederate-1_ping-cloud_pingfederate", "logEvents": [{"id": "37013848024578310180644099321922695591001628886545793024", "timestamp": 1659758348659, "message": {"log": "/opt/out/instance/log/server.log 2022-08-06 03:59:02,867 tid:qf8Wb6tGdzy8PN4tl93rHepNMR0 DEBUG [org.sourceid.websso.servlet.IntegrationControllerServlet] GET: https://localhost:9031/pf/heartbeat.ping\n", "stream": "stdout", "docker": {"container_id": "1f9796fb6abf364e5be93bf7bda4242ad75d987b80141f19ff62a2ddbe7ac2ce"}, "kubernetes": {"container_name": "pingfederate", "namespace_name": "ping-cloud", "pod_name": "pingfederate-1", "container_image": "public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate:10.3.5-v1.0.23-no-IKs", "container_image_id": "docker-pullable://public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate@sha256:b781191a0a206d4779e4959c7f0cc14ec9a8022692a0481bbf38438dad49a7be", "pod_id": "278d7ef7-a6c1-47d0-a65f-7f5ad711fcd8", "host": "ip-10-10-117-227.us-east-2.compute.internal", "labels": {"app": "ping-cloud", "cluster": "pingfederate-cluster", "controller-revision-hash": "pingfederate-75b6bfc8fd", "role": "pingfederate-engine", "statefulset_kubernetes_io/pod-name": "pingfederate-1"}, "master_url": "https://172.20.0.1:443/api", "namespace_id": "38da143e-d00d-489a-9284-560673491f1d", "namespace_labels": {"app": "ping-cloud", "app_kubernetes_io/instance": "ping-cloud-master-us-east-2", "kubernetes_io/metadata_name": "ping-cloud"}}, "stream_name": "pingfederate-1_ping-cloud_pingfederate"}}, {"id": "37013848024578310180644099321922695591001628886545793025", "timestamp": 1659758348659, "message": {"log": "/opt/out/instance/log/server.log 2022-08-06 03:59:02,868 tid:qf8Wb6tGdzy8PN4tl93rHepNMR0 DEBUG [org.sourceid.servlet.HttpServletRespProxy] flush cookies: adding Cookie{PF=hashedValue:qf8Wb6tGdzy8PN4tl93rHepNMR0; path=/; maxAge=-1; domain=null}\n", "stream": "stdout", "docker": {"container_id": "1f9796fb6abf364e5be93bf7bda4242ad75d987b80141f19ff62a2ddbe7ac2ce"}, "kubernetes": {"container_name": "pingfederate", "namespace_name": "ping-cloud", "pod_name": "pingfederate-1", "container_image": "public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate:10.3.5-v1.0.23-no-IKs", "container_image_id": "docker-pullable://public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate@sha256:b781191a0a206d4779e4959c7f0cc14ec9a8022692a0481bbf38438dad49a7be", "pod_id": "278d7ef7-a6c1-47d0-a65f-7f5ad711fcd8", "host": "ip-10-10-117-227.us-east-2.compute.internal", "labels": {"app": "ping-cloud", "cluster": "pingfederate-cluster", "controller-revision-hash": "pingfederate-75b6bfc8fd", "role": "pingfederate-engine", "statefulset_kubernetes_io/pod-name": "pingfederate-1"}, "master_url": "https://172.20.0.1:443/api", "namespace_id": "38da143e-d00d-489a-9284-560673491f1d", "namespace_labels": {"app": "ping-cloud", "app_kubernetes_io/instance": "ping-cloud-master-us-east-2", "kubernetes_io/metadata_name": "ping-cloud"}}, "stream_name": "pingfederate-1_ping-cloud_pingfederate"}}, {"id": "37013848024578310180644099321922695591001628886545793026", "timestamp": 1659758348659, "message": {"log": "/opt/out/instance/log/server.log 2022-08-06 03:59:07,871 DEBUG [org.sourceid.util.log.internal.TrackingIdSupport] The incoming request does not contain a unique identifier. Assigning auto-generated request ID: OdFVgBFxp1JYNoiCQenljYo0I\n", "stream": "stdout", "docker": {"container_id": "1f9796fb6abf364e5be93bf7bda4242ad75d987b80141f19ff62a2ddbe7ac2ce"}, "kubernetes": {"container_name": "pingfederate", "namespace_name": "ping-cloud", "pod_name": "pingfederate-1", "container_image": "public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate:10.3.5-v1.0.23-no-IKs", "container_image_id": "docker-pullable://public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate@sha256:b781191a0a206d4779e4959c7f0cc14ec9a8022692a0481bbf38438dad49a7be", "pod_id": "278d7ef7-a6c1-47d0-a65f-7f5ad711fcd8", "host": "ip-10-10-117-227.us-east-2.compute.internal", "labels": {"app": "ping-cloud", "cluster": "pingfederate-cluster", "controller-revision-hash": "pingfederate-75b6bfc8fd", "role": "pingfederate-engine", "statefulset_kubernetes_io/pod-name": "pingfederate-1"}, "master_url": "https://172.20.0.1:443/api", "namespace_id": "38da143e-d00d-489a-9284-560673491f1d", "namespace_labels": {"app": "ping-cloud", "app_kubernetes_io/instance": "ping-cloud-master-us-east-2", "kubernetes_io/metadata_name": "ping-cloud"}}, "stream_name": "pingfederate-1_ping-cloud_pingfederate"}}, {"id": "37013848085437043827434169875173670757059007436366348291", "timestamp": 1659758351388, "message": {"log": "/opt/out/instance/log/server.log 2022-08-06 03:59:07,871 DEBUG [org.sourceid.servlet.HttpServletRespProxy] adding lazy cookie Cookie{PF=hashedValue:VizAUYH0x9Lu7GbN_Rqv9fevZ7c; path=/; maxAge=-1; domain=null} replacing null\n", "stream": "stdout", "docker": {"container_id": "1f9796fb6abf364e5be93bf7bda4242ad75d987b80141f19ff62a2ddbe7ac2ce"}, "kubernetes": {"container_name": "pingfederate", "namespace_name": "ping-cloud", "pod_name": "pingfederate-1", "container_image": "public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate:10.3.5-v1.0.23-no-IKs", "container_image_id": "docker-pullable://public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate@sha256:b781191a0a206d4779e4959c7f0cc14ec9a8022692a0481bbf38438dad49a7be", "pod_id": "278d7ef7-a6c1-47d0-a65f-7f5ad711fcd8", "host": "ip-10-10-117-227.us-east-2.compute.internal", "labels": {"app": "ping-cloud", "cluster": "pingfederate-cluster", "controller-revision-hash": "pingfederate-75b6bfc8fd", "role": "pingfederate-engine", "statefulset_kubernetes_io/pod-name": "pingfederate-1"}, "master_url": "https://172.20.0.1:443/api", "namespace_id": "38da143e-d00d-489a-9284-560673491f1d", "namespace_labels": {"app": "ping-cloud", "app_kubernetes_io/instance": "ping-cloud-master-us-east-2", "kubernetes_io/metadata_name": "ping-cloud"}}, "stream_name": "pingfederate-1_ping-cloud_pingfederate"}}, {"id": "37013848091480545776235968746529850408946713404487041028", "timestamp": 1659758351659, "message": {"log": "/opt/out/instance/log/server.log 2022-08-06 03:59:07,871 tid:VizAUYH0x9Lu7GbN_Rqv9fevZ7c DEBUG [org.sourceid.websso.servlet.IntegrationControllerServlet] GET: https://localhost:9031/pf/heartbeat.ping\n", "stream": "stdout", "docker": {"container_id": "1f9796fb6abf364e5be93bf7bda4242ad75d987b80141f19ff62a2ddbe7ac2ce"}, "kubernetes": {"container_name": "pingfederate", "namespace_name": "ping-cloud", "pod_name": "pingfederate-1", "container_image": "public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate:10.3.5-v1.0.23-no-IKs", "container_image_id": "docker-pullable://public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate@sha256:b781191a0a206d4779e4959c7f0cc14ec9a8022692a0481bbf38438dad49a7be", "pod_id": "278d7ef7-a6c1-47d0-a65f-7f5ad711fcd8", "host": "ip-10-10-117-227.us-east-2.compute.internal", "labels": {"app": "ping-cloud", "cluster": "pingfederate-cluster", "controller-revision-hash": "pingfederate-75b6bfc8fd", "role": "pingfederate-engine", "statefulset_kubernetes_io/pod-name": "pingfederate-1"}, "master_url": "https://172.20.0.1:443/api", "namespace_id": "38da143e-d00d-489a-9284-560673491f1d", "namespace_labels": {"app": "ping-cloud", "app_kubernetes_io/instance": "ping-cloud-master-us-east-2", "kubernetes_io/metadata_name": "ping-cloud"}}, "stream_name": "pingfederate-1_ping-cloud_pingfederate"}}, {"id": "37013848091480545776235968746529850408946713404487041029", "timestamp": 1659758351659, "message": {"log": "/opt/out/instance/log/server.log 2022-08-06 03:59:07,871 tid:VizAUYH0x9Lu7GbN_Rqv9fevZ7c DEBUG [org.sourceid.servlet.HttpServletRespProxy] flush cookies: adding Cookie{PF=hashedValue:VizAUYH0x9Lu7GbN_Rqv9fevZ7c; path=/; maxAge=-1; domain=null}\n", "stream": "stdout", "docker": {"container_id": "1f9796fb6abf364e5be93bf7bda4242ad75d987b80141f19ff62a2ddbe7ac2ce"}, "kubernetes": {"container_name": "pingfederate", "namespace_name": "ping-cloud", "pod_name": "pingfederate-1", "container_image": "public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate:10.3.5-v1.0.23-no-IKs", "container_image_id": "docker-pullable://public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate@sha256:b781191a0a206d4779e4959c7f0cc14ec9a8022692a0481bbf38438dad49a7be", "pod_id": "278d7ef7-a6c1-47d0-a65f-7f5ad711fcd8", "host": "ip-10-10-117-227.us-east-2.compute.internal", "labels": {"app": "ping-cloud", "cluster": "pingfederate-cluster", "controller-revision-hash": "pingfederate-75b6bfc8fd", "role": "pingfederate-engine", "statefulset_kubernetes_io/pod-name": "pingfederate-1"}, "master_url": "https://172.20.0.1:443/api", "namespace_id": "38da143e-d00d-489a-9284-560673491f1d", "namespace_labels": {"app": "ping-cloud", "app_kubernetes_io/instance": "ping-cloud-master-us-east-2", "kubernetes_io/metadata_name": "ping-cloud"}}, "stream_name": "pingfederate-1_ping-cloud_pingfederate"}}, {"id": "37013848091480545776235968746529850408946713404487041030", "timestamp": 1659758351659, "message": {"log": "/opt/out/instance/log/server.log 2022-08-06 03:59:11,388 DEBUG [org.sourceid.util.log.internal.TrackingIdSupport] The incoming request does not contain a unique identifier. Assigning auto-generated request ID: ssP8SFNvaJjisaEtFgH9aIN3q\n", "stream": "stdout", "docker": {"container_id": "1f9796fb6abf364e5be93bf7bda4242ad75d987b80141f19ff62a2ddbe7ac2ce"}, "kubernetes": {"container_name": "pingfederate", "namespace_name": "ping-cloud", "pod_name": "pingfederate-1", "container_image": "public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate:10.3.5-v1.0.23-no-IKs", "container_image_id": "docker-pullable://public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate@sha256:b781191a0a206d4779e4959c7f0cc14ec9a8022692a0481bbf38438dad49a7be", "pod_id": "278d7ef7-a6c1-47d0-a65f-7f5ad711fcd8", "host": "ip-10-10-117-227.us-east-2.compute.internal", "labels": {"app": "ping-cloud", "cluster": "pingfederate-cluster", "controller-revision-hash": "pingfederate-75b6bfc8fd", "role": "pingfederate-engine", "statefulset_kubernetes_io/pod-name": "pingfederate-1"}, "master_url": "https://172.20.0.1:443/api", "namespace_id": "38da143e-d00d-489a-9284-560673491f1d", "namespace_labels": {"app": "ping-cloud", "app_kubernetes_io/instance": "ping-cloud-master-us-east-2", "kubernetes_io/metadata_name": "ping-cloud"}}, "stream_name": "pingfederate-1_ping-cloud_pingfederate"}}, {"id": "37013848118464447466458022747788069518851230826723344391", "timestamp": 1659758352869, "message": {"log": "/opt/out/instance/log/server.log 2022-08-06 03:59:11,388 DEBUG [org.sourceid.servlet.HttpServletRespProxy] adding lazy cookie Cookie{PF=hashedValue:hFl_yLtHqOydi68KkvReugURSyc; path=/; maxAge=-1; domain=null} replacing null\n", "stream": "stdout", "docker": {"container_id": "1f9796fb6abf364e5be93bf7bda4242ad75d987b80141f19ff62a2ddbe7ac2ce"}, "kubernetes": {"container_name": "pingfederate", "namespace_name": "ping-cloud", "pod_name": "pingfederate-1", "container_image": "public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate:10.3.5-v1.0.23-no-IKs", "container_image_id": "docker-pullable://public.ecr.aws/r2h3l6e4/pingcloud-apps/pingfederate@sha256:b781191a0a206d4779e4959c7f0cc14ec9a8022692a0481bbf38438dad49a7be", "pod_id": "278d7ef7-a6c1-47d0-a65f-7f5ad711fcd8", "host": "ip-10-10-117-227.us-east-2.compute.internal", "labels": {"app": "ping-cloud", "cluster": "pingfederate-cluster", "controller-revision-hash": "pingfederate-75b6bfc8fd", "role": "pingfederate-engine", "statefulset_kubernetes_io/pod-name": "pingfederate-1"}, "master_url": "https://172.20.0.1:443/api", "namespace_id": "38da143e-d00d-489a-9284-560673491f1d", "namespace_labels": {"app": "ping-cloud", "app_kubernetes_io/instance": "ping-cloud-master-us-east-2", "kubernetes_io/metadata_name": "ping-cloud"}}, "stream_name": "pingfederate-1_ping-cloud_pingfederate"}}], "envType": "prod"}

[richgalloway]

When multiple events are combined it's often because the LINE_BREAKER setting is incorrect.  Verify the setting in your props.conf file.  If you need help then share the props.conf settings here.