All Apps and Add-ons

Splunk Kubernetes Create index by Namespace

rmurali4u
New Member

Hi
Little background
I have a EKS cluster and On Premise splunk cluster. We have 5-10 application micro-service running on EKS.
I want ingest logs into the Splunk from EKS K8.
Splunk Connector has been configured and able to ingest logs.

At present all the K8 pods logs converged into splunk-objects pods and object pod logs are mapped to one index (kube_obj-index)in the splunk.

index="kube_obj-index" namespace="myapplication1" "GET" | collect index=myapplication1-logs
index="kube_obj-index" namespace="myapplication2" "GET" | collect index=myapplication2-logs
index="kube_obj-index" namespace="myapplication3" "GET" | collect index=myapplication3-logs

I need to help to modify yaml files so that i want each Kubernetes Namespace logs goes to separate index in the Splunk.

0 Karma

mattymo
Splunk Employee
Splunk Employee

have you solved this, or do you still need help? Our dev branch has annotation support in preview now as well.

- MattyMo
0 Karma

rmurali4u
New Member

@richgalloway
Sorry I don't see your comment. Would you please post it again.Thanks

0 Karma

maciep
Champion

does the Namespace to Index Routing section on the readme help?

https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/README.md

0 Karma

rmurali4u
New Member

@maciep Thanks for your comment. It worked

0 Karma

rmurali4u
New Member

This is the information

For example: Consider the following kubernetes namespace to splunk index topology.
(Namespace) -> (Splunk Index)
kube-system -> kube-system
kube-public -> kube-public
default -> indexRoutingDefaultIndex For this topology to work appropriately we have to create the splunk indexes "kube-system", "kube-public" and the value of indexRoutingDefaultIndex.

I have created below indexes
kube-system
kube-public
namespace1
namespace2

HEC_TOKEN is mapped to an index in the splunk. I used that index as a default index.
I am getting below error.

19-09-17 17:20:47 +0000 [info]: #0 fluentd worker is now running worker=0
19-09-17 17:20:53 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:00 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:07 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:14 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:21 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:28 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:34 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}

Any thought and do you think i am doing anything wrong.

Thanks.

0 Karma

maciep
Champion

Are you specifying "allowed" indexes for that token on the HEC. I think that's when I've seen the "Incorrect index" problem before - the index you're sending to isn't in the allowed list.

I think typically the approach is to not define any allowed indexes and rely on the default index to catch unexpected data.

0 Karma

rmurali4u
New Member

@maciep

Thanks I will try and let you know.

0 Karma

vaibhav
New Member

Hello @rmurali4u I'm running into the same issue. I'm running 1.4.7 version connector. Any inputs would be helpful.

0 Karma
Get Updates on the Splunk Community!

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...