New Splunker here! Need help with event breaking!

[oliverb_pickles]

Hello all, 

I do apologise as I am a new Splunker and needing some help with event breaking. Not sure the best approach as my raw data is unreadable. 

What is the best method for parsing the log with field extractions + line/event breaking. 

Here is an example of a log:

{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/dsadsa/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Metadata\",\"auditID\":\"f07ee314-89e5-4743-a515-05f18dfd1c32\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/api/v1/namespaces/nginx-activate-account-test/configmaps/ingress-controller-leader-nginx-activate-account-test\",\"verb\":\"update\",\"user\":{\"username\":\"system:serviceaccount:nginx-activate-account-test:nginx-activate-account-test-nginx-ingress\",\"uid\":\"7490dbfe-63ea-4c65-b79c-dc9975e1996a\",\"groups\":[\"system:serviceaccounts\",\"system:serviceaccounts:nginx-activate-account-test\",\"system:authenticated\"]},\"sourceIPs\":[\"10.241.0.23\"],\"userAgent\":\"nginx-ingress-controller/v0.34.1 (linux/amd64) ingress-nginx/v20200715-ingress-nginx-2.11.0-8-gda5fa45e2\",\"objectRef\":{\"resource\":\"configmaps\",\"namespace\":\"nginx-activate-account-test\",\"name\":\"ingress-controller-leader-nginx-activate-account-test\",\"uid\":\"072c4bc7-a841-458e-af05-9b98e0d80724\",\"apiVersion\":\"v1\",\"resourceVersion\":\"77895625\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-25T22:07:43.638765Z\",\"stageTimestamp\":\"2021-04-25T22:07:43.641341Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by RoleBinding \\\"nginx-activate-account-test-nginx-ingress/nginx-activate-account-test\\\" of Role \\\"nginx-activate-account-test-nginx-ingress\\\" to ServiceAccount \\\"nginx-activate-account-test-nginx-ingress/nginx-activate-account-test\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-25T22:07:43.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"}

[gcusello]

Hi @oliverb_pickles,

this seems to be a json format, in this case use the 

INDEXED_EXTRACTIONS = JSON

in props.conf, as described at https://docs.splunk.com/Documentation/Splunk/8.1.3/Data/Extractfieldsfromfileswithstructureddata

In this way you already have all the ield automatically extracted.

The best approach is that you put a sample of your logs in a file and follow the web guided Add data procedure so you can create your props.conf.

Ciao.

Giuseppe

[oliverb_pickles]

Hello Giuseppe,

Thanks for the response. I did try to add this in the props.conf but the format still came out quite messy. It actually only formatted sections of the raw data into JSON and the rest was still raw.

I did however start to explore line breaking and was able to break up events. But the format is not what I was hoping for. 

Please see screenshot.

what i am hoping for is something like this below:

@gcusello 

[gcusello]

Hi @oliverb_pickles,

the problem is that you divided your events in more events and so Splunk doesn't recognize json format, try to see if you can break your events in a different way.

At the same time, put attention that each event isn't truncated.

If you reach to have each event with a full json event you'll have the desiderated format.

To help you I need a sample of your logs (a full event) and your props.conf.

Ciao.

Giuseppe

[oliverb_pickles]

{"body":{"records": [{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"cebf788f-345c-45cd-818d-d534ba42970c\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/apis/apps/v1/namespaces/nginx-identityserver-admin-dev/deployments/nginx-identityserver-admin-dev-nginx-ingress-default-backend/scale\",\"verb\":\"get\",\"user\":{\"username\":\"system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"uid\":\"903bb43c-633c-48d0-b3d4-b32676041353\",\"groups\":[\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"]},\"sourceIPs\":[\"172.31.30.46\"],\"userAgent\":\"kube-controller-manager/v1.17.11 (linux/amd64) kubernetes/3a36121/system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"objectRef\":{\"resource\":\"deployments\",\"namespace\":\"nginx-identityserver-admin-dev\",\"name\":\"nginx-identityserver-admin-dev-nginx-ingress-default-backend\",\"apiGroup\":\"apps\",\"apiVersion\":\"v1\",\"subresource\":\"scale\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-28T01:12:00.045180Z\",\"stageTimestamp\":\"2021-04-28T01:12:00.048161Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:controller:horizontal-pod-autoscaler\\\" of ClusterRole \\\"system:controller:horizontal-pod-autoscaler\\\" to ServiceAccount \\\"horizontal-pod-autoscaler/kube-system\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-28T01:12:00.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"},{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"a7d90b81-e355-4c01-95cb-c7c61c08a58c\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/api/v1/namespaces/monitoring/persistentvolumeclaims/prometheus-server\",\"verb\":\"get\",\"user\":{\"username\":\"nodeclient\",\"groups\":[\"system:nodes\",\"system:authenticated\"]},\"sourceIPs\":[\"10.241.0.23\"],\"userAgent\":\"kubelet/v1.17.11 (linux/amd64) kubernetes/3a36121\",\"objectRef\":{\"resource\":\"persistentvolumeclaims\",\"namespace\":\"monitoring\",\"name\":\"prometheus-server\",\"apiVersion\":\"v1\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-28T01:12:00.121676Z\",\"stageTimestamp\":\"2021-04-28T01:12:00.123980Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:aks-client-nodes\\\" of ClusterRole \\\"system:node\\\" to Group \\\"system:nodes\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-28T01:12:00.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"},{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"8dc487c7-7ff5-4201-a1a4-a77ad84f047a\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/api/v1/persistentvolumes/pvc-04485553-7b85-484e-a872-d4c3a7e54380\",\"verb\":\"get\",\"user\":{\"username\":\"nodeclient\",\"groups\":[\"system:nodes\",\"system:authenticated\"]},\"sourceIPs\":[\"10.241.0.23\"],\"userAgent\":\"kubelet/v1.17.11 (linux/amd64) kubernetes/3a36121\",\"objectRef\":{\"resource\":\"persistentvolumes\",\"name\":\"pvc-04485553-7b85-484e-a872-d4c3a7e54380\",\"apiVersion\":\"v1\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-28T01:12:00.125450Z\",\"stageTimestamp\":\"2021-04-28T01:12:00.127476Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:aks-client-nodes\\\" of ClusterRole \\\"system:node\\\" to Group \\\"system:nodes\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-28T01:12:00.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"},{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"6c4e2cd4-05c7-4f4b-8e13-2d4b28369b7f\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/apis/metrics.k8s.io/v1beta1/namespaces/nginx-identityserver-admin-dev/pods?labelSelector=app%3Dnginx-ingress%2Crelease%3Dnginx-identityserver-admin-dev\",\"verb\":\"list\",\"user\":{\"username\":\"system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"uid\":\"903bb43c-633c-48d0-b3d4-b32676041353\",\"groups\":[\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"]},\"sourceIPs\":[\"172.31.30.46\"],\"userAgent\":\"kube-controller-manager/v1.17.11 (linux/amd64) kubernetes/3a36121/system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"objectRef\":{\"resource\":\"pods\",\"namespace\":\"nginx-identityserver-admin-dev\",\"apiGroup\":\"metrics.k8s.io\",\"apiVersion\":\"v1beta1\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-28T01:12:00.049877Z\",\"stageTimestamp\":\"2021-04-28T01:12:00.141528Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:controller:horizontal-pod-autoscaler\\\" of ClusterRole \\\"system:controller:horizontal-pod-autoscaler\\\" to ServiceAccount \\\"horizontal-pod-autoscaler/kube-system\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-28T01:12:00.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"},{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"90c4c427-0136-4f29-b922-a3cd0d44762c\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/apis/metrics.k8s.io/v1beta1/namespaces/nginx-identityserver-admin-dev/pods?labelSelector=app%3Dnginx-ingress%2Crelease%3Dnginx-identityserver-admin-dev\",\"verb\":\"list\",\"user\":{\"username\":\"system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"uid\":\"903bb43c-633c-48d0-b3d4-b32676041353\",\"groups\":[\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"]},\"sourceIPs\":[\"172.31.30.46\"],\"userAgent\":\"kube-controller-manager/v1.17.11 (linux/amd64) kubernetes/3a36121/system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"objectRef\":{\"resource\":\"pods\",\"namespace\":\"nginx-identityserver-admin-dev\",\"apiGroup\":\"metrics.k8s.io\",\"apiVersion\":\"v1beta1\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-28T01:12:00.143315Z\",\"stageTimestamp\":\"2021-04-28T01:12:00.221061Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:controller:horizontal-pod-autoscaler\\\" of ClusterRole \\\"system:controller:horizontal-pod-autoscaler\\\" to ServiceAccount \\\"horizontal-pod-autoscaler/kube-system\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-28T01:12:00.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"},{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"e83f20d1-d039-4d57-b414-4cb92eec5a38\",\"stage\":\"ResponseStarted\",\"requestURI\":\"/api/v1/nodes/

[oliverb_pickles]

Please see raw data above and here is the props.conf below:

INDEXED_EXTRACTIONS = JSON
SHOULD_LINEMERGE = false
NO_BINARY_CHECK =true
LINE_BREAKER = ([\r\n]*)"operationName"

[gcusello]

Hi @oliverb_pickles,

if you break your event at every operationName, you haven't more a json.

Try to leave the event entire (don't use the LINE_BREAKER option, in this way you shold have a complete json event.

I see that you's event is very long, I think that's better to add an option TRUNCATE = 1000000 to avoid truncating.

Ciao.

Giuseppe

[oliverb_pickles]

@gcusello 

Thank you so much. Please see new props.conf config below:

INDEXED_EXTRACTIONS = JSON
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true
TRUNCATE = 1000000

---

I do have a few more questions if you could please assist.

1. The logs appear in a JSON format however each event isn't breaking. Please see screenshot below. I understand each event has the same time stamp but is it possible to still to break them into individual events?

2. Under the log section, there are further sub fields I would like to extract and have highlighted.. Is this possible?

3. How do I hide fields which are not relevant? e.g. Steam is not useful