Archive
Highlighted

Splunk Kubernetes Create index by Namespace

New Member

Hi
Little background
I have a EKS cluster and On Premise splunk cluster. We have 5-10 application micro-service running on EKS.
I want ingest logs into the Splunk from EKS K8.
Splunk Connector has been configured and able to ingest logs.

At present all the K8 pods logs converged into splunk-objects pods and object pod logs are mapped to one index (kube_obj-index)in the splunk.

index="kubeobj-index" namespace="myapplication1" "GET" | collect index=myapplication1-logs
index="kube
obj-index" namespace="myapplication2" "GET" | collect index=myapplication2-logs
index="kube_obj-index" namespace="myapplication3" "GET" | collect index=myapplication3-logs

I need to help to modify yaml files so that i want each Kubernetes Namespace logs goes to separate index in the Splunk.

0 Karma
Highlighted

Re: Splunk Kubernetes Create index by Namespace

Champion

does the Namespace to Index Routing section on the readme help?

https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/README.md

0 Karma
Highlighted

Re: Splunk Kubernetes Create index by Namespace

New Member

@maciep

Thanks I will try and let you know.

0 Karma
Highlighted

Re: Splunk Kubernetes Create index by Namespace

New Member

This is the information

For example: Consider the following kubernetes namespace to splunk index topology.
(Namespace) -> (Splunk Index)
kube-system -> kube-system
kube-public -> kube-public
default -> indexRoutingDefaultIndex For this topology to work appropriately we have to create the splunk indexes "kube-system", "kube-public" and the value of indexRoutingDefaultIndex.

I have created below indexes
kube-system
kube-public
namespace1
namespace2

HEC_TOKEN is mapped to an index in the splunk. I used that index as a default index.
I am getting below error.

19-09-17 17:20:47 +0000 [info]: #0 fluentd worker is now running worker=0
19-09-17 17:20:53 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:00 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:07 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:14 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:21 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:28 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:34 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}

Any thought and do you think i am doing anything wrong.

Thanks.

0 Karma
Highlighted

Re: Splunk Kubernetes Create index by Namespace

Champion

Are you specifying "allowed" indexes for that token on the HEC. I think that's when I've seen the "Incorrect index" problem before - the index you're sending to isn't in the allowed list.

I think typically the approach is to not define any allowed indexes and rely on the default index to catch unexpected data.

0 Karma
Highlighted

Re: Splunk Kubernetes Create index by Namespace

New Member

@maciep Thanks for your comment. It worked

0 Karma
Highlighted

Re: Splunk Kubernetes Create index by Namespace

New Member

@richgalloway
Sorry I don't see your comment. Would you please post it again.Thanks

0 Karma
Highlighted

Re: Splunk Kubernetes Create index by Namespace

Splunk Employee
Splunk Employee

have you solved this, or do you still need help? Our dev branch has annotation support in preview now as well.

0 Karma