Splunk Search

how to route the data based on event filed

syaseensplunk
Engager

Hi,

I'm running a test setup with some live kubernetes data and I want to do the following indexer:

1) Route all data matching a certain field to a specific index called "gsp" on my indexer.


I already have been playing around with the _MetaData:Index key which seems to work just fine when applied as single transform for a certain sourcetype. However, How I have multiple sourcetypes.

This is my props.conf

[kube:container*]
TRANSFORMS-routing = AnthosGSP

This is my transforms.conf

[AnthosGSP]
REGEX = drnt0-retail-sabbnetservices
DEST_KEY = _MetaData:Index
FORMAT = gsp

However, the routing isn't happening as it should be.Please help!!

PS: I am a newbie to splunking.. so pardon my ignorance.

Regards,

Yaseen.

Labels (1)
0 Karma

syaseensplunk
Engager

{"severity":"INFO","ts":1704101563.224535,"logger":"controller","msg":"Seccomp profile 'not configured' is not allowed for container 'splunk-fluentd-k8s-objects'. Found at: no explicit profile found. Allowed profiles: {\"RuntimeDefault\", \"docker/default\", \"runtime/default\"}","process":"audit","audit_id":"2024-01-01T09:32:31Z","details":{},"event_type":"violation_audited","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sPSPSeccomp","constraint_name":"cis-k8s-v1.5.1-psp-seccomp-default","constraint_namespace":"","constraint_action":"warn","resource_group":"","resource_api_version":"v1","resource_kind":"Pod","resource_namespace":"idmzct0-ito-utils-splunkdc-callsign","resource_name":"gkeusr-idmzc-dev-tier0-01-splunk-kubernetes-objects-5686d96j7nj","resource_labels":{"app":"splunk-kubernetes-objects","engine":"fluentd","pod-template-hash":"5686d96bd8","release":"gkeusr-idmzc-dev-tier0-01"}}
Show syntax highlighted
cluster_name = gkeusr-idmzc-dev-tier0-01container_name = managerhost = npool-cos-apps-medium-7b7dd5cdb8-s6lrpnamespace = gatekeeper-systempod = gatekeeper-audit-789888c597-q9vt8severity = INFOsource = /var/log/containers/gatekeeper-audit-789888c597-q9vt8_gatekeeper-system_manager-da5f687a6b53035c4299f8e3c5cc941c510756de883f2f0e68e783cd4edc7191.logsourcetype = kube:container:manager

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @syaseensplunk,

if this is a sample of the logs to filter, the regex in the transforms.conf doesn't match any event.

You have to use a different regex tat has a match with the events, e.g. something like this:

REGEX = \"app\":\"splunk-kubernetes-objects\"

or a different one, that you can test at https://regex101.com/r/GnkJqh/1 

Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @syaseensplunk,

at first: where do you locate the conf files?

they must be located in the Heavy Forwarder that you're using to take logs from Kubernetes or in the first Full Splunk instance that the data pass through.

Second question:

are you sure that the regex that you inserted in transforms.conf matches the events to override index?

Ciao.

Giuseppe

0 Karma

syaseensplunk
Engager

Thanks for the response.. @Giuseppe.

where do you locate the conf files?

The conf files are located at the first full splunk instance that the data pass through.

Regarding the REGEX, what I am trying to achieve is data to be routed to the specified index in transforms.conf based on the field name and it's value.

In this case, what I am looking for is whenever, there is a <namespace="drnt0-retail-sabbnetservices"> in the data I want the routing to work.

Regards,

Yaseen.

 

Regards,

Yaseen.

 

 

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @syaseensplunk,

ok for the regex.

But where do you located the conf files?

if there's another Fulls Splunk instance (an Heavy Forwarder) before the location of conf files, they don't work.

Ciao.

Giuseppe

0 Karma

syaseensplunk
Engager

There is none.. However, I was able to make it work with the <source_type>.

Any help is much appreciated!!

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @syaseensplunk,

It's always better to use sourcetype, alsop because I'm not sure that you can use the kubernetes contaiener, you can only use sourcetype, host and source.

Sorry if I ask you again: where do you located these conf files?

Ciao.

Giuseppe

0 Karma

syaseensplunk
Engager

My files are located on the indexer/indexer's/cluster-master under "/opt/splunk/etc/apps/appName/local".

Yes, it works with sourcetype. However, it seems the sourcetype spec doesn't accept wildcard.

[kube:container:*] - is there a way I can make it work? I need every source with "kube:container:<container_name>" to be accepted in props.conf

 

Secondly, in my transforms.conf , I want to route any event with "namespace="drnt0-retail-sabbnetservices"" to my already existing index created separately to receive this events data. - Please help me with this. 

 

Regards,

Yaseen.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @syaseensplunk,

As I supposed, probably the issue is the location of the conf files: they must be in the first full Splunk instance they pass throgh.

In other words, in the Heavy Forwarder (if present) used to extract logs from Kubernetes or in the Indexers, not installed in the Cluster Master.

If you have to install them in the Indexers, you have to use the Cluster Master to deploy them to the Indexers, but not installing in the folder you said, you have to copy them in $SPLUNK_HOME/etc/manager-apps and deploy them as described at https://docs.splunk.com/Documentation/Splunk/9.1.2/Indexer/Manageappdeployment .

Ciao.

Giuseppe

0 Karma

syaseensplunk
Engager

There are no heavy forwarders.. .

Below is the summary of things for your understanding..

I've successfully configured Splunk Connect for Kubernetes and are ingesting data into the "events" index.
I'd like to redirect this data into more meaningful indexes based on specific field values, such as the "namespace" field.
I've been able to achieve rerouting using sourcetype configurations in props.conf and transforms.conf.
But using other fields like "namespace" configuration in transform.conf and props.conf file, log data is not redirected to other meaning full indexes.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @syaseensplunk,

as I said, for my knowledge in props.conf you can use only sourcetype or source or host, not kubernetes namespace.

And syntax is the following:

sourcetype

[mysourcetype]

source:

[source::my_source]

host:

[host::my_host]

Ciao.

Giuseppe

0 Karma

syaseensplunk
Engager

Is there a way I can use "namespace" in transforms.conf to seperate the data after using sourcetype in props.conf

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @syaseensplunk,

in transforms.conf, you put a transformation called from the props.conf, so you can put whatever you want:

# props.conf
[your_sourcetype]
TRANSFORMS-routing = AnthosGSP

#transforma.conf
[AnthosGSP]
REGEX = drnt0-retail-sabbnetservices
DEST_KEY = _MetaData:Index
FORMAT = gsp

Ciao.

Giuseppe

0 Karma

syaseensplunk
Engager

the same stanzan's is what I had been trying, but no luck 😞

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @syaseensplunk,

let me better understand your architecture, correct me if I'm wrong:

  • you have a Universal Forwarder in your kubernetes or do you take logs using API from an heavy Forwarders?
  • does this Universal or Heavy Forwarder send logs to an intermediate Heavy Forwarder or directly to an Indexers?

Please describe the flow of your data from kubernetes to the Indexers in terms of machines.

Ciao.

Giuseppe

0 Karma

syaseensplunk
Engager

I have splunk connect in kubernetes which is responsible for forwarding the logs directly to the indexers using HEC token. Hope this helps!!

the props.conf and transforms.conf should be on the indexer layer to process the incoming data from Kubernetes via splunk connect. - this is my understanding

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @syaseensplunk,

yes, it's correct, the location is on Indexers, even if I don't like to have te inputs directly on Indexers, I prefer to have a dedicated Heavy Forwarder (better two with a Load Balancer for HA),

so coming beck to your issue, it's anoter one:

could you share sample of your logs, to check the regex?

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...