Getting Data In

New Splunker here! Need help with event breaking!

oliverb_pickles
Loves-to-Learn

Hello all, 

I do apologise as I am a new Splunker and needing some help with event breaking. Not sure the best approach as my raw data is unreadable. 

What is the best method for parsing the log with field extractions + line/event breaking. 

Here is an example of a log:

 

 

{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/dsadsa/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Metadata\",\"auditID\":\"f07ee314-89e5-4743-a515-05f18dfd1c32\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/api/v1/namespaces/nginx-activate-account-test/configmaps/ingress-controller-leader-nginx-activate-account-test\",\"verb\":\"update\",\"user\":{\"username\":\"system:serviceaccount:nginx-activate-account-test:nginx-activate-account-test-nginx-ingress\",\"uid\":\"7490dbfe-63ea-4c65-b79c-dc9975e1996a\",\"groups\":[\"system:serviceaccounts\",\"system:serviceaccounts:nginx-activate-account-test\",\"system:authenticated\"]},\"sourceIPs\":[\"10.241.0.23\"],\"userAgent\":\"nginx-ingress-controller/v0.34.1 (linux/amd64) ingress-nginx/v20200715-ingress-nginx-2.11.0-8-gda5fa45e2\",\"objectRef\":{\"resource\":\"configmaps\",\"namespace\":\"nginx-activate-account-test\",\"name\":\"ingress-controller-leader-nginx-activate-account-test\",\"uid\":\"072c4bc7-a841-458e-af05-9b98e0d80724\",\"apiVersion\":\"v1\",\"resourceVersion\":\"77895625\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-25T22:07:43.638765Z\",\"stageTimestamp\":\"2021-04-25T22:07:43.641341Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by RoleBinding \\\"nginx-activate-account-test-nginx-ingress/nginx-activate-account-test\\\" of Role \\\"nginx-activate-account-test-nginx-ingress\\\" to ServiceAccount \\\"nginx-activate-account-test-nginx-ingress/nginx-activate-account-test\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-25T22:07:43.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"}

 

 

 

0 Karma

gcusello
Legend

Hi @oliverb_pickles,

this seems to be a json format, in this case use the 

INDEXED_EXTRACTIONS = JSON

in props.conf, as described at https://docs.splunk.com/Documentation/Splunk/8.1.3/Data/Extractfieldsfromfileswithstructureddata

In this way you already have all the ield automatically extracted.

The best approach is that you put a sample of your logs in a file and follow the web guided Add data procedure so you can create your props.conf.

Ciao.

Giuseppe

0 Karma

oliverb_pickles
Loves-to-Learn

Hello Giuseppe,

Thanks for the response. I did try to add this in the props.conf but the format still came out quite messy. It actually only formatted sections of the raw data into JSON and the rest was still raw.

 

I did however start to explore line breaking and was able to break up events. But the format is not what I was hoping for. 

Please see screenshot.

 


what i am hoping for is something like this below:

oliverb_pickles_0-1619612711294.png

 

 
 

@gcusello 

0 Karma

gcusello
Legend

Hi @oliverb_pickles,

the problem is that you divided your events in more events and so Splunk doesn't recognize json format, try to see if you can break your events in a different way.

At the same time, put attention that each event isn't truncated.

If you reach to have each event with a full json event you'll have the desiderated format.

To help you I need a sample of your logs (a full event) and your props.conf.

Ciao.

Giuseppe

0 Karma

oliverb_pickles
Loves-to-Learn
{"body":{"records": [{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"cebf788f-345c-45cd-818d-d534ba42970c\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/apis/apps/v1/namespaces/nginx-identityserver-admin-dev/deployments/nginx-identityserver-admin-dev-nginx-ingress-default-backend/scale\",\"verb\":\"get\",\"user\":{\"username\":\"system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"uid\":\"903bb43c-633c-48d0-b3d4-b32676041353\",\"groups\":[\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"]},\"sourceIPs\":[\"172.31.30.46\"],\"userAgent\":\"kube-controller-manager/v1.17.11 (linux/amd64) kubernetes/3a36121/system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"objectRef\":{\"resource\":\"deployments\",\"namespace\":\"nginx-identityserver-admin-dev\",\"name\":\"nginx-identityserver-admin-dev-nginx-ingress-default-backend\",\"apiGroup\":\"apps\",\"apiVersion\":\"v1\",\"subresource\":\"scale\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-28T01:12:00.045180Z\",\"stageTimestamp\":\"2021-04-28T01:12:00.048161Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:controller:horizontal-pod-autoscaler\\\" of ClusterRole \\\"system:controller:horizontal-pod-autoscaler\\\" to ServiceAccount \\\"horizontal-pod-autoscaler/kube-system\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-28T01:12:00.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"},{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"a7d90b81-e355-4c01-95cb-c7c61c08a58c\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/api/v1/namespaces/monitoring/persistentvolumeclaims/prometheus-server\",\"verb\":\"get\",\"user\":{\"username\":\"nodeclient\",\"groups\":[\"system:nodes\",\"system:authenticated\"]},\"sourceIPs\":[\"10.241.0.23\"],\"userAgent\":\"kubelet/v1.17.11 (linux/amd64) kubernetes/3a36121\",\"objectRef\":{\"resource\":\"persistentvolumeclaims\",\"namespace\":\"monitoring\",\"name\":\"prometheus-server\",\"apiVersion\":\"v1\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-28T01:12:00.121676Z\",\"stageTimestamp\":\"2021-04-28T01:12:00.123980Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:aks-client-nodes\\\" of ClusterRole \\\"system:node\\\" to Group \\\"system:nodes\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-28T01:12:00.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"},{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"8dc487c7-7ff5-4201-a1a4-a77ad84f047a\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/api/v1/persistentvolumes/pvc-04485553-7b85-484e-a872-d4c3a7e54380\",\"verb\":\"get\",\"user\":{\"username\":\"nodeclient\",\"groups\":[\"system:nodes\",\"system:authenticated\"]},\"sourceIPs\":[\"10.241.0.23\"],\"userAgent\":\"kubelet/v1.17.11 (linux/amd64) kubernetes/3a36121\",\"objectRef\":{\"resource\":\"persistentvolumes\",\"name\":\"pvc-04485553-7b85-484e-a872-d4c3a7e54380\",\"apiVersion\":\"v1\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-28T01:12:00.125450Z\",\"stageTimestamp\":\"2021-04-28T01:12:00.127476Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:aks-client-nodes\\\" of ClusterRole \\\"system:node\\\" to Group \\\"system:nodes\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-28T01:12:00.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"},{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"6c4e2cd4-05c7-4f4b-8e13-2d4b28369b7f\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/apis/metrics.k8s.io/v1beta1/namespaces/nginx-identityserver-admin-dev/pods?labelSelector=app%3Dnginx-ingress%2Crelease%3Dnginx-identityserver-admin-dev\",\"verb\":\"list\",\"user\":{\"username\":\"system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"uid\":\"903bb43c-633c-48d0-b3d4-b32676041353\",\"groups\":[\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"]},\"sourceIPs\":[\"172.31.30.46\"],\"userAgent\":\"kube-controller-manager/v1.17.11 (linux/amd64) kubernetes/3a36121/system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"objectRef\":{\"resource\":\"pods\",\"namespace\":\"nginx-identityserver-admin-dev\",\"apiGroup\":\"metrics.k8s.io\",\"apiVersion\":\"v1beta1\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-28T01:12:00.049877Z\",\"stageTimestamp\":\"2021-04-28T01:12:00.141528Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:controller:horizontal-pod-autoscaler\\\" of ClusterRole \\\"system:controller:horizontal-pod-autoscaler\\\" to ServiceAccount \\\"horizontal-pod-autoscaler/kube-system\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-28T01:12:00.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"},{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"90c4c427-0136-4f29-b922-a3cd0d44762c\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/apis/metrics.k8s.io/v1beta1/namespaces/nginx-identityserver-admin-dev/pods?labelSelector=app%3Dnginx-ingress%2Crelease%3Dnginx-identityserver-admin-dev\",\"verb\":\"list\",\"user\":{\"username\":\"system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"uid\":\"903bb43c-633c-48d0-b3d4-b32676041353\",\"groups\":[\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"]},\"sourceIPs\":[\"172.31.30.46\"],\"userAgent\":\"kube-controller-manager/v1.17.11 (linux/amd64) kubernetes/3a36121/system:serviceaccount:kube-system:horizontal-pod-autoscaler\",\"objectRef\":{\"resource\":\"pods\",\"namespace\":\"nginx-identityserver-admin-dev\",\"apiGroup\":\"metrics.k8s.io\",\"apiVersion\":\"v1beta1\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2021-04-28T01:12:00.143315Z\",\"stageTimestamp\":\"2021-04-28T01:12:00.221061Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:controller:horizontal-pod-autoscaler\\\" of ClusterRole \\\"system:controller:horizontal-pod-autoscaler\\\" to ServiceAccount \\\"horizontal-pod-autoscaler/kube-system\\\"\"}}\n","stream":"stdout","pod":"kube-apiserver-64bc7458dc-nhccb"}, "time": "2021-04-28T01:12:00.0000000Z", "Cloud": "AzureCloud", "Environment": "prod", "UnderlayClass": "hcp-underlay", "UnderlayName": "hcp-underlay-australiaeast-cx-36"},{ "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category": "kube-audit", "ccpNamespace": "5fc0e650f40a0500013bfedc", "resourceId": "/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/P365-AUE-MGMT-DTA-FRONTEND-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/P365-AUE-MGMT-DTA-FRONTEND-AKS", "properties": {"log":"{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Request\",\"auditID\":\"e83f20d1-d039-4d57-b414-4cb92eec5a38\",\"stage\":\"ResponseStarted\",\"requestURI\":\"/api/v1/nodes/
0 Karma

oliverb_pickles
Loves-to-Learn

Please see raw data above and here is the props.conf below:

 

INDEXED_EXTRACTIONS = JSON
SHOULD_LINEMERGE = false
NO_BINARY_CHECK =true
LINE_BREAKER = ([\r\n]*)"operationName"
0 Karma

gcusello
Legend

Hi @oliverb_pickles,

if you break your event at every operationName, you haven't more a json.

Try to leave the event entire (don't use the LINE_BREAKER option, in this way you shold have a complete json event.

I see that you's event is very long, I think that's better to add an option TRUNCATE = 1000000 to avoid truncating.

Ciao.

Giuseppe

0 Karma

oliverb_pickles
Loves-to-Learn

@gcusello 

Thank you so much. Please see new props.conf config below:

INDEXED_EXTRACTIONS = JSON
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true
TRUNCATE = 1000000

---

I do have a few more questions if you could please assist.

1. The logs appear in a JSON format however each event isn't breaking. Please see screenshot below. I understand each event has the same time stamp but is it possible to still to break them into individual events?

2. Under the log section, there are further sub fields I would like to extract and have highlighted.. Is this possible?

3. How do I hide fields which are not relevant? e.g. Steam is not useful

oliverb_pickles_0-1619694223780.png

 

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!