how to route the data based on event filed

[syaseensplunk]

Hi,

I'm running a test setup with some live kubernetes data and I want to do the following indexer:

1) Route all data matching a certain field to a specific index called "gsp" on my indexer.

I already have been playing around with the _MetaData:Index key which seems to work just fine when applied as single transform for a certain sourcetype. However, How I have multiple sourcetypes.

This is my props.conf

[kube:container*]
TRANSFORMS-routing = AnthosGSP

This is my transforms.conf

[AnthosGSP]
REGEX = drnt0-retail-sabbnetservices
DEST_KEY = _MetaData:Index
FORMAT = gsp

However, the routing isn't happening as it should be.Please help!!

PS: I am a newbie to splunking.. so pardon my ignorance.

Regards,

Yaseen.

[syaseensplunk]

{"severity":"INFO","ts":1704101563.224535,"logger":"controller","msg":"Seccomp profile 'not configured' is not allowed for container 'splunk-fluentd-k8s-objects'. Found at: no explicit profile found. Allowed profiles: {\"RuntimeDefault\", \"docker/default\", \"runtime/default\"}","process":"audit","audit_id":"2024-01-01T09:32:31Z","details":{},"event_type":"violation_audited","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sPSPSeccomp","constraint_name":"cis-k8s-v1.5.1-psp-seccomp-default","constraint_namespace":"","constraint_action":"warn","resource_group":"","resource_api_version":"v1","resource_kind":"Pod","resource_namespace":"idmzct0-ito-utils-splunkdc-callsign","resource_name":"gkeusr-idmzc-dev-tier0-01-splunk-kubernetes-objects-5686d96j7nj","resource_labels":{"app":"splunk-kubernetes-objects","engine":"fluentd","pod-template-hash":"5686d96bd8","release":"gkeusr-idmzc-dev-tier0-01"}}
Show syntax highlighted
cluster_name = gkeusr-idmzc-dev-tier0-01container_name = managerhost = npool-cos-apps-medium-7b7dd5cdb8-s6lrpnamespace = gatekeeper-systempod = gatekeeper-audit-789888c597-q9vt8severity = INFOsource = /var/log/containers/gatekeeper-audit-789888c597-q9vt8_gatekeeper-system_manager-da5f687a6b53035c4299f8e3c5cc941c510756de883f2f0e68e783cd4edc7191.logsourcetype = kube:container:manager

[gcusello]

Hi @syaseensplunk,

if this is a sample of the logs to filter, the regex in the transforms.conf doesn't match any event.

You have to use a different regex tat has a match with the events, e.g. something like this:

REGEX = \"app\":\"splunk-kubernetes-objects\"

or a different one, that you can test at https://regex101.com/r/GnkJqh/1 

Ciao.

Giuseppe

[gcusello]

Hi @syaseensplunk,

at first: where do you locate the conf files?

they must be located in the Heavy Forwarder that you're using to take logs from Kubernetes or in the first Full Splunk instance that the data pass through.

Second question:

are you sure that the regex that you inserted in transforms.conf matches the events to override index?

Ciao.

Giuseppe

[syaseensplunk]

Thanks for the response.. @Giuseppe.

where do you locate the conf files?

The conf files are located at the first full splunk instance that the data pass through.

Regarding the REGEX, what I am trying to achieve is data to be routed to the specified index in transforms.conf based on the field name and it's value.

In this case, what I am looking for is whenever, there is a <namespace="drnt0-retail-sabbnetservices"> in the data I want the routing to work.

Regards,

Yaseen.

Regards,

Yaseen.

[gcusello]

Hi @syaseensplunk,

ok for the regex.

But where do you located the conf files?

if there's another Fulls Splunk instance (an Heavy Forwarder) before the location of conf files, they don't work.

Ciao.

Giuseppe

[syaseensplunk]

There is none.. However, I was able to make it work with the <source_type>.

Any help is much appreciated!!

[gcusello]

Hi @syaseensplunk,

It's always better to use sourcetype, alsop because I'm not sure that you can use the kubernetes contaiener, you can only use sourcetype, host and source.

Sorry if I ask you again: where do you located these conf files?

Ciao.

Giuseppe

[syaseensplunk]

My files are located on the indexer/indexer's/cluster-master under "/opt/splunk/etc/apps/appName/local".

Yes, it works with sourcetype. However, it seems the sourcetype spec doesn't accept wildcard.

[kube:container:*] - is there a way I can make it work? I need every source with "kube:container:<container_name>" to be accepted in props.conf

Secondly, in my transforms.conf , I want to route any event with "namespace="drnt0-retail-sabbnetservices"" to my already existing index created separately to receive this events data. - Please help me with this. 

Regards,

Yaseen.

[gcusello]

Hi @syaseensplunk,

As I supposed, probably the issue is the location of the conf files: they must be in the first full Splunk instance they pass throgh.

In other words, in the Heavy Forwarder (if present) used to extract logs from Kubernetes or in the Indexers, not installed in the Cluster Master.

If you have to install them in the Indexers, you have to use the Cluster Master to deploy them to the Indexers, but not installing in the folder you said, you have to copy them in $SPLUNK_HOME/etc/manager-apps and deploy them as described at https://docs.splunk.com/Documentation/Splunk/9.1.2/Indexer/Manageappdeployment .

Ciao.

Giuseppe

[syaseensplunk]

There are no heavy forwarders.. .

Below is the summary of things for your understanding..

I've successfully configured Splunk Connect for Kubernetes and are ingesting data into the "events" index.
I'd like to redirect this data into more meaningful indexes based on specific field values, such as the "namespace" field.
I've been able to achieve rerouting using sourcetype configurations in props.conf and transforms.conf.
But using other fields like "namespace" configuration in transform.conf and props.conf file, log data is not redirected to other meaning full indexes.

[gcusello]

Hi @syaseensplunk,

as I said, for my knowledge in props.conf you can use only sourcetype or source or host, not kubernetes namespace.

And syntax is the following:

sourcetype

[mysourcetype]

source:

[source::my_source]

host:

[host::my_host]

Ciao.

Giuseppe

[syaseensplunk]

Is there a way I can use "namespace" in transforms.conf to seperate the data after using sourcetype in props.conf

[gcusello]

Hi @syaseensplunk,

in transforms.conf, you put a transformation called from the props.conf, so you can put whatever you want:

# props.conf
[your_sourcetype]
TRANSFORMS-routing = AnthosGSP

#transforma.conf
[AnthosGSP]
REGEX = drnt0-retail-sabbnetservices
DEST_KEY = _MetaData:Index
FORMAT = gsp

Ciao.

Giuseppe

[syaseensplunk]

the same stanzan's is what I had been trying, but no luck 😞

[gcusello]

Hi @syaseensplunk,

let me better understand your architecture, correct me if I'm wrong:

• you have a Universal Forwarder in your kubernetes or do you take logs using API from an heavy Forwarders?
• does this Universal or Heavy Forwarder send logs to an intermediate Heavy Forwarder or directly to an Indexers?

Please describe the flow of your data from kubernetes to the Indexers in terms of machines.

Ciao.

Giuseppe

[syaseensplunk]

I have splunk connect in kubernetes which is responsible for forwarding the logs directly to the indexers using HEC token. Hope this helps!!

the props.conf and transforms.conf should be on the indexer layer to process the incoming data from Kubernetes via splunk connect. - this is my understanding

[gcusello]

Hi @syaseensplunk,

yes, it's correct, the location is on Indexers, even if I don't like to have te inputs directly on Indexers, I prefer to have a dedicated Heavy Forwarder (better two with a Load Balancer for HA),

so coming beck to your issue, it's anoter one:

could you share sample of your logs, to check the regex?

Ciao.

Giuseppe