Splunk Kubernetes Create index by Namespace

[rmurali4u]

Hi
Little background
I have a EKS cluster and On Premise splunk cluster. We have 5-10 application micro-service running on EKS.
I want ingest logs into the Splunk from EKS K8.
Splunk Connector has been configured and able to ingest logs.

At present all the K8 pods logs converged into splunk-objects pods and object pod logs are mapped to one index (kube_obj-index)in the splunk.

index="kube_obj-index" namespace="myapplication1" "GET" | collect index=myapplication1-logs
index="kube_obj-index" namespace="myapplication2" "GET" | collect index=myapplication2-logs
index="kube_obj-index" namespace="myapplication3" "GET" | collect index=myapplication3-logs

I need to help to modify yaml files so that i want each Kubernetes Namespace logs goes to separate index in the Splunk.

[mattymo]

have you solved this, or do you still need help? Our dev branch has annotation support in preview now as well.

[rmurali4u]

@richgalloway
Sorry I don't see your comment. Would you please post it again.Thanks

[maciep]

does the Namespace to Index Routing section on the readme help?

https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/README.md

[rmurali4u]

@maciep Thanks for your comment. It worked

[rmurali4u]

This is the information

For example: Consider the following kubernetes namespace to splunk index topology.
(Namespace) -> (Splunk Index)
kube-system -> kube-system
kube-public -> kube-public
default -> indexRoutingDefaultIndex For this topology to work appropriately we have to create the splunk indexes "kube-system", "kube-public" and the value of indexRoutingDefaultIndex.

I have created below indexes
kube-system
kube-public
namespace1
namespace2

HEC_TOKEN is mapped to an index in the splunk. I used that index as a default index.
I am getting below error.

19-09-17 17:20:47 +0000 [info]: #0 fluentd worker is now running worker=0
19-09-17 17:20:53 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:00 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:07 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:14 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:21 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:28 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:34 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}

Any thought and do you think i am doing anything wrong.

Thanks.

[maciep]

Are you specifying "allowed" indexes for that token on the HEC. I think that's when I've seen the "Incorrect index" problem before - the index you're sending to isn't in the allowed list.

I think typically the approach is to not define any allowed indexes and rely on the default index to catch unexpected data.

[rmurali4u]

@maciep

Thanks I will try and let you know.

[vaibhav]

Hello @rmurali4u I'm running into the same issue. I'm running 1.4.7 version connector. Any inputs would be helpful.