topic Re: Need help not indexing some specific events that match a field in Getting Data In

How to not index some specific events that match a field?

dritjon — Tue, 19 Jul 2022 16:14:07 GMT

Because of licensing reasons, I want to stop indexing these events (as they make up almost 50% of the index)

index=cisco dest_port=53

So basically DNS requests. Is it possible for this specific index=cisco to stop indexing these logs where dest_port=53? I cant do it from the cisco firewall itself.

I googled a bit and the consensus seems to be sending the logs to NULLQUEUE, and modify props.conf & transform.conf. But what I'm struggling with is where are these files?

My Splunk architecture is 2 Search Heads in a cluster and 1 License Manager server. Where to modify these files? On both Search heads?

Re: Need help not indexing some specific events that match a field

richgalloway — Tue, 19 Jul 2022 13:59:48 GMT

There can be many props.conf and transforms.conf files in a Splunk instance. You'll find them in $SPLUNK_HOME/etc/system/default, $SPLUNK_HOME/etc/system/local, $SPLUNK_HOME/etc/apps/<appname>/default, and $SPLUNK_HOME/etc/apps/<appname>/local (ignoring user-specific files). Splunk combines them all, using precedence rules, to produce a run-time configuration.

Never modify a .conf file in a default directory. Any such changes will be lost the next time Splunk or the app is upgraded.

Where do you make your changes? In the app that defines the sourcetype being modified. That may be a Cisco add-on or a custom app.

Your architecture seems unusual. A search head cluster is supposed to have at least 3 search heads and you don't mention indexers at all. The settings to send unwanted events to the null queue must be installed on each indexer. If you don't have separate indexers then the settings go on the SHs.

Re: Need help not indexing some specific events that match a field

somesoni2 — Tue, 19 Jul 2022 14:03:03 GMT

The data routing props/transforms are setup on node where data is parsed and usually it's the indexers where that happens. If you're using Heavy forwarder (a node with Splunk Enterprise on it and does the data collection), then the data parsing happens on heavy forwarder.

Also, the null routing happens based on sourcetype/source/host and not index. So identify which sourcetypes/source/host are sending events with dest=53, write a regex which will run on _raw (raw data) and setup appropriate configurations for filtering out the data before indexing.

https://docs.splunk.com/Documentation/SplunkCloud/8.2.2203/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues

Re: Need help not indexing some specific events that match a field

dritjon — Tue, 19 Jul 2022 14:18:17 GMT

@richgalloway wrote:
There can be many props.conf and transforms.conf files in a Splunk instance. You'll find them in $SPLUNK_HOME/etc/system/default, $SPLUNK_HOME/etc/system/local, $SPLUNK_HOME/etc/apps/<appname>/default, and $SPLUNK_HOME/etc/apps/<appname>/local (ignoring user-specific files). Splunk combines them all, using precedence rules, to produce a run-time configuration.
Never modify a .conf file in a default directory. Any such changes will be lost the next time Splunk or the app is upgraded.
Where do you make your changes? In the app that defines the sourcetype being modified. That may be a Cisco add-on or a custom app.
Your architecture seems unusual. A search head cluster is supposed to have at least 3 search heads and you don't mention indexers at all. The settings to send unwanted events to the null queue must be installed on each indexer. If you don't have separate indexers then the settings go on the SHs.

Sorry as I'm new to splunk. I have 1 search head and 2 indexers. Do I need to change the files on the search head or indexer? My /opt path on both machines has these folders splunkforwarder, splunk_indexer, syslog

Re: Need help not indexing some specific events that match a field

dritjon — Tue, 19 Jul 2022 14:12:47 GMT

The source is /opt/syslog/10.101.132.1/

The sourcetype is cisco:asa

My architecture has 2 indexers in a cluster.

Do i have to edit the files on both indexers

Re: Need help not indexing some specific events that match a field

somesoni2 — Tue, 19 Jul 2022 14:15:52 GMT

Yes, since both indexers can index data and parse it, it should be on both.

Since they're clustered, you could create an app containing those configuration and deploy it from Cluster Manager/master. See this: https://docs.splunk.com/Documentation/Splunk/9.0.0/Indexer/Updatepeerconfigurations

Re: Need help not indexing some specific events that match a field

dritjon — Tue, 19 Jul 2022 14:31:16 GMT

Thanks. One last question.

The official doc says to modify the file in this path

$SPLUNK_HOME/etc/system/local/props.conf

But my local path doesnt have a props.conf file. Instead the path

$SPLUNK_HOME/etc/system/default/

has a props.conf file

Which to update?

Re: Need help not indexing some specific events that match a field

richgalloway — Tue, 19 Jul 2022 14:52:49 GMT

As I mentioned earlier, NEVER MODIFY A FILE IN A default DIRECTORY.

If the file does not exist in local then create it.

Re: Need help not indexing some specific events that match a field

richgalloway — Tue, 19 Jul 2022 14:58:50 GMT

As @somesoni2 and I said, the changes should be done on the indexer(s).

Re: Need help not indexing some specific events that match a field

somesoni2 — Tue, 19 Jul 2022 16:10:49 GMT

I would say create a new app on Cluster Manager/Master ($SPLUNK_HOME/etc/master-apps/), say cisco_routing_props_transforms and create file "cisco_routing_props_transforms/local/props.conf" and "cisco_routing_props_transforms/local/transforms.conf". After that deploy the app to both indexer cluster peer. That way both indexers will always have same config.