@gcusello  Thank you so much. Please see new props.conf config below: INDEXED_EXTRACTIONS = JSON SHOULD_LINEMERGE = false NO_BINARY_CHECK = true TRUNCATE = 1000000 --- I do have a few more questions if you could please assist. 1. The logs appear in a JSON format however each event isn't breaking. Please see screenshot below. I understand each event has the same time stamp but is it possible to still to break them into individual events? 2. Under the log section, there are further sub fields I would like to extract and have highlighted.. Is this possible? 3. How do I hide fields which are not relevant? e.g. Steam is not useful   ... View more

Please see raw data above and here is the props.conf below:   INDEXED_EXTRACTIONS = JSON SHOULD_LINEMERGE = false NO_BINARY_CHECK =true LINE_BREAKER = ([\r\n]*)"operationName" ... View more

{"body":{"records": [{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"cebf788f-345c-45cd-818d-d534ba42970c\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/apis/apps/v1/namespaces/nginx-identityserver-admin-dev/deployments/nginx-identityserver-admin-dev-nginx-ingress-default-backend/scale\",\"verb\":\"get\",\"user\":{\"username\":\"system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"uid\":\"903bb43c-633c-48d0-b3d4-b32676041353\",\"groups\":[\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"]},\"sourceIPs\":[\"172.31.30.46\"],\"userAgent\":\"kube-controller-manager/v1.17.11 (linux/amd64) kubernetes/3a36121/system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"objectRef\":{\"resource\":\"deployments\",\"namespace\":\"nginx-identityserver-admin-dev\",\"name\":\"nginx-identityserver-admin-dev-nginx-ingress-default-backend\",\"apiGroup\":\"apps\",\"apiVersion\":\"v1\",\"subresource\":\"scale\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-28T01:12:00.045180Z\",\"stageTimestamp\":\"2021-04-28T01:12:00.048161Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:controller:horizontal-pod-autoscaler\\\" of ClusterRole \\\"system:controller:horizontal-pod-autoscaler\\\" to ServiceAccount \\\"horizontal-pod-autoscaler/kube-system\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-28T01:12:00.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"},{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"a7d90b81-e355-4c01-95cb-c7c61c08a58c\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/api/v1/namespaces/monitoring/persistentvolumeclaims/prometheus-server\",\"verb\":\"get\",\"user\":{\"username\":\"nodeclient\",\"groups\":[\"system:nodes\",\"system:authenticated\"]},\"sourceIPs\":[\"10.241.0.23\"],\"userAgent\":\"kubelet/v1.17.11 (linux/amd64) kubernetes/3a36121\",\"objectRef\":{\"resource\":\"persistentvolumeclaims\",\"namespace\":\"monitoring\",\"name\":\"prometheus-server\",\"apiVersion\":\"v1\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-28T01:12:00.121676Z\",\"stageTimestamp\":\"2021-04-28T01:12:00.123980Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:aks-client-nodes\\\" of ClusterRole \\\"system:node\\\" to Group \\\"system:nodes\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-28T01:12:00.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"},{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"8dc487c7-7ff5-4201-a1a4-a77ad84f047a\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/api/v1/persistentvolumes/pvc-04485553-7b85-484e-a872-d4c3a7e54380\",\"verb\":\"get\",\"user\":{\"username\":\"nodeclient\",\"groups\":[\"system:nodes\",\"system:authenticated\"]},\"sourceIPs\":[\"10.241.0.23\"],\"userAgent\":\"kubelet/v1.17.11 (linux/amd64) kubernetes/3a36121\",\"objectRef\":{\"resource\":\"persistentvolumes\",\"name\":\"pvc-04485553-7b85-484e-a872-d4c3a7e54380\",\"apiVersion\":\"v1\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-28T01:12:00.125450Z\",\"stageTimestamp\":\"2021-04-28T01:12:00.127476Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:aks-client-nodes\\\" of ClusterRole \\\"system:node\\\" to Group \\\"system:nodes\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-28T01:12:00.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"},{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"6c4e2cd4-05c7-4f4b-8e13-2d4b28369b7f\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/apis/metrics.k8s.io/v1beta1/namespaces/nginx-identityserver-admin-dev/pods?labelSelector=app%3Dnginx-ingress%2Crelease%3Dnginx-identityserver-admin-dev\",\"verb\":\"list\",\"user\":{\"username\":\"system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"uid\":\"903bb43c-633c-48d0-b3d4-b32676041353\",\"groups\":[\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"]},\"sourceIPs\":[\"172.31.30.46\"],\"userAgent\":\"kube-controller-manager/v1.17.11 (linux/amd64) kubernetes/3a36121/system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"objectRef\":{\"resource\":\"pods\",\"namespace\":\"nginx-identityserver-admin-dev\",\"apiGroup\":\"metrics.k8s.io\",\"apiVersion\":\"v1beta1\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-28T01:12:00.049877Z\",\"stageTimestamp\":\"2021-04-28T01:12:00.141528Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:controller:horizontal-pod-autoscaler\\\" of ClusterRole \\\"system:controller:horizontal-pod-autoscaler\\\" to ServiceAccount \\\"horizontal-pod-autoscaler/kube-system\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-28T01:12:00.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"},{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"90c4c427-0136-4f29-b922-a3cd0d44762c\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/apis/metrics.k8s.io/v1beta1/namespaces/nginx-identityserver-admin-dev/pods?labelSelector=app%3Dnginx-ingress%2Crelease%3Dnginx-identityserver-admin-dev\",\"verb\":\"list\",\"user\":{\"username\":\"system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"uid\":\"903bb43c-633c-48d0-b3d4-b32676041353\",\"groups\":[\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"]},\"sourceIPs\":[\"172.31.30.46\"],\"userAgent\":\"kube-controller-manager/v1.17.11 (linux/amd64) kubernetes/3a36121/system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"objectRef\":{\"resource\":\"pods\",\"namespace\":\"nginx-identityserver-admin-dev\",\"apiGroup\":\"metrics.k8s.io\",\"apiVersion\":\"v1beta1\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-28T01:12:00.143315Z\",\"stageTimestamp\":\"2021-04-28T01:12:00.221061Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:controller:horizontal-pod-autoscaler\\\" of ClusterRole \\\"system:controller:horizontal-pod-autoscaler\\\" to ServiceAccount \\\"horizontal-pod-autoscaler/kube-system\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-28T01:12:00.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"},{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"e83f20d1-d039-4d57-b414-4cb92eec5a38\",\"stage\":\"ResponseStarted\",\"requestURI\":\"/api/v1/nodes/ ... View more

Hello Giuseppe, Thanks for the response. I did try to add this in the props.conf but the format still came out quite messy. It actually only formatted sections of the raw data into JSON and the rest was still raw.   I did however start to explore line breaking and was able to break up events. But the format is not what I was hoping for.  Please see screenshot.   what i am hoping for is something like this below:       @gcusello  ... View more