topic Re: Splunk Kubernetes Create index by Namespace in All Apps and Add-ons

Splunk Kubernetes Create index by Namespace

rmurali4u — Wed, 30 Sep 2020 02:08:09 GMT

Hi
Little background
I have a EKS cluster and On Premise splunk cluster. We have 5-10 application micro-service running on EKS.
I want ingest logs into the Splunk from EKS K8.
Splunk Connector has been configured and able to ingest logs.

At present all the K8 pods logs converged into splunk-objects pods and object pod logs are mapped to one index (kube_obj-index)in the splunk.

index="kube_obj-index" namespace="myapplication1" "GET" | collect index=myapplication1-logs
index="kube_obj-index" namespace="myapplication2" "GET" | collect index=myapplication2-logs
index="kube_obj-index" namespace="myapplication3" "GET" | collect index=myapplication3-logs

I need to help to modify yaml files so that i want each Kubernetes Namespace logs goes to separate index in the Splunk.

Re: Splunk Kubernetes Create index by Namespace

maciep — Sat, 14 Sep 2019 15:07:25 GMT

does the Namespace to Index Routing section on the readme help?

https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/README.md

Re: Splunk Kubernetes Create index by Namespace

rmurali4u — Tue, 17 Sep 2019 16:49:37 GMT

@maciep

Thanks I will try and let you know.

Re: Splunk Kubernetes Create index by Namespace

rmurali4u — Tue, 17 Sep 2019 16:50:53 GMT

@richgalloway
Sorry I don't see your comment. Would you please post it again.Thanks

Re: Splunk Kubernetes Create index by Namespace

rmurali4u — Tue, 17 Sep 2019 17:58:41 GMT

This is the information

For example: Consider the following kubernetes namespace to splunk index topology.
(Namespace) -> (Splunk Index)
kube-system -> kube-system
kube-public -> kube-public
default -> indexRoutingDefaultIndex For this topology to work appropriately we have to create the splunk indexes "kube-system", "kube-public" and the value of indexRoutingDefaultIndex.

I have created below indexes
kube-system
kube-public
namespace1
namespace2

HEC_TOKEN is mapped to an index in the splunk. I used that index as a default index.
I am getting below error.

19-09-17 17:20:47 +0000 [info]: #0 fluentd worker is now running worker=0
19-09-17 17:20:53 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:00 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:07 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:14 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:21 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:28 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:34 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}

Any thought and do you think i am doing anything wrong.

Thanks.

Re: Splunk Kubernetes Create index by Namespace

maciep — Tue, 17 Sep 2019 19:04:24 GMT

Are you specifying "allowed" indexes for that token on the HEC. I think that's when I've seen the "Incorrect index" problem before - the index you're sending to isn't in the allowed list.

I think typically the approach is to not define any allowed indexes and rely on the default index to catch unexpected data.

Re: Splunk Kubernetes Create index by Namespace

rmurali4u — Fri, 06 Dec 2019 19:04:28 GMT

@maciep Thanks for your comment. It worked

Re: Splunk Kubernetes Create index by Namespace

mattymo — Wed, 29 Jan 2020 16:53:55 GMT

have you solved this, or do you still need help? Our dev branch has annotation support in preview now as well.

Re: Splunk Kubernetes Create index by Namespace

vaibhav — Tue, 04 May 2021 01:13:28 GMT

Hello @rmurali4u I'm running into the same issue. I'm running 1.4.7 version connector. Any inputs would be helpful.