topic drop events from being indexed in Getting Data In

drop events from being indexed

sreynolds30 — Wed, 11 Apr 2018 14:18:47 GMT

I have a request from some users of mine to do the following.

I need to drop events from a source and user ..

source: /var/log/uds/uds.log
user: dsapi_perftest

Re: drop events from being indexed

logloganathan — Wed, 11 Apr 2018 15:25:01 GMT

Could you please try this query

source= /var/log/uds/uds.log NOT "dsapi_perftest"

it will produce the event without the user from a source you mentioned

Re: drop events from being indexed

niketn — Wed, 11 Apr 2018 16:00:58 GMT

@sreynolds30, one of the options you have is to search the data to be made unsearchable and run the delete command. You have to be aware that it will only make the data unsearchable and not remove from storage. Read about the delete command and understand its usage before applying.

Also before you delete existing data, you should also make sure that source uds.log is not sending data for user dsapi_perftest. If it is you should apply Regular Expression to filter out the event. Refer to documentation to filter data and send unwanted events to nullQueue before indexing.

Re: drop events from being indexed

sreynolds30 — Wed, 11 Apr 2018 18:01:56 GMT

Sorry I guess i should have stated this better... that i just want to drop this events from being indexed and leave everything else within that source.

I'll look at the unwanted events to nullQueue

Re: drop events from being indexed

sreynolds30 — Wed, 11 Apr 2018 18:02:46 GMT

Sorry I guess i should have stated this better... that i just want to drop this events from being indexed and leave everything else within that source.

I'll look at the unwanted events to nullQueue

Re: drop events from being indexed

niketn — Wed, 11 Apr 2018 18:44:48 GMT

@sreynolds30, nullQueue will drop future events from being indexed however, you delete command was a suggestion for clearing out existing events for the user which are already indexed. Even if you do not delete, they would age out based on your index bucket rollover policy/size.

Please try out nullQueue and confirm whether you need further assistance.

Re: drop events from being indexed

sreynolds30 — Tue, 29 Sep 2020 18:58:48 GMT

@niketn Thanks for the input.

I'm working on the nullQueue in a different test but it's not working as i think it should. Here's a sample of the logs that i don't want to index from my client from this source but just for that user.

2018-04-11T08:49:34,140 1077.dti.net [UDS] http-nio-8080-exec-25 (com.uds.logging.RequestLogger) INFO - env="UDS-US-Stable" requestid="bWGDWVP7FJMNhjD@awAAAC8" user="dsapi_perftest" tenantid="625c-bd25-47a5-a12d-4db5396d4ceb" role="dsapi_perftest" appid="b6de5707-cbee-477b-a332-98edce3e38e9" appname="qdxmmldbpubk" requests="1000" records="1000" status="200" method="POST" URL="/uds/batch/match/"
2018-04-11T08:49:34,353 1077.dti.net [UDS] http-nio-8080-exec-46 (com.uds.logging.RequestLogger) INFO - env="UDS-US-Stable" requestid="bmGDWVP7FJMNhjD@bAAAAC8" user="dsapi_perftest" tenantid="625c-bd25-47a5-a12d-4db5396d4ceb" role="dsapi_perftest" appid="b6de5707-cbee-477b-a332-98edce3e38e9" appname="qdxmmldbpubk" requests="1000" records="1000" status="200" method="POST" URL="/uds/batch/match/"
2018-04-11T08:49:34,552 1077.dti.net [UDS] http-nio-8080-exec-173 (com.uds.logging.RequestLogger) INFO - env="UDS-US-Stable" requestid="bmGDWVP7FJMNhjD@bQAAAC8" user="dsapi_perftest" tenantid="625c-bd25-47a5-a12d-4db5396d4ceb" role="dsapi_perftest" appid="b6de5707-cbee-477b-a332-98edce3e38e9" appname="qdxmmldbpubk" requests="1000" records="1000" status="200" method="POST" URL="/uds/batch/match/"
2018-04-11T08:49:34,763 1077.dti.net [UDS] http-nio-8080-exec-236 (com.uds.logging.RequestLogger) INFO - env="UDS-US-Stable" requestid="bmGDWVP7FJMNhjD@bgAAAC8" user="dsapi_perftest" tenantid="625c-bd25-47a5-a12d-4db5396d4ceb" role="dsapi_perftest" appid="b6de5707-cbee-477b-a332-98edce3e38e9" appname="qdxmmldbpubk" requests="1000" records="1000" status="200" method="POST" URL="/uds/batch/match/"
2018-04-11T08:49:34,989 1077.dti.net [UDS] http-nio-8080-exec-157 (com.uds.logging.RequestLogger) INFO - env="UDS-US-Stable" requestid="bmGDWVP7FJMNhjD@bwAAAC8" user="dsapi_perftest" tenantid="625c-bd25-47a5-a12d-4db5396d4ceb" role="dsapi_perftest" appid="b6de5707-cbee-477b-a332-98edce3e38e9" appname="qdxmmldbpubk" requests="1000" records="1000" status="200" method="POST" URL="/uds/batch/match/"
2018-04-11T08:49:35,196 1077.dti.net [UDS] http-nio-8080-exec-180 (com.uds.logging.RequestLogger) INFO - env="UDS-US-Stable" requestid="b2GDWVP7FJMNhjD@cAAAAC8" user="dsapi_perftest" tenantid="625c-bd25-47a5-a12d-4db5396d4ceb" role="dsapi_perftest" appid="b6de5707-cbee-477b-a332-98edce3e38e9" appname="qdxmmldbpubk" requests="1000" records="1000" status="200" method="POST" URL="/uds/batch/match/"

Re: drop events from being indexed

niketn — Thu, 12 Apr 2018 05:33:03 GMT

@sreynolds30, have you tried the configurations on the following line?

props.conf

[yourSourceType]
TRANSFORMS-nullQueueUnwantedUser = nullQueueUnwantedUser

transforms.conf

[nullQueueUnwantedUser]
REGEX = user\=\"dsapi_perftest\"
DEST_KEY = queue
FORMAT = nullQueue

Test using Splunk's _internal index whether events are getting dropped or not:

index=_internal sourcetype=splunkd source=*metrics.log component=metrics group=pipeline processor=nullqueue

Also, events can be dropped on indexers or Heavy Forwarders, not on Universal Forwarder.

Re: drop events from being indexed

sreynolds30 — Tue, 17 Apr 2018 19:47:51 GMT

I got it working. Thanks for the feedback @niketnilay

Re: drop events from being indexed

niketn — Tue, 17 Apr 2018 19:54:42 GMT

@sreynolds30, glad you got it to work. I have converted my comment to answer. Accept to mark this as answered and upvote the comments that helped.

Re: drop events from being indexed

bhargavi — Fri, 12 Nov 2021 08:00:52 GMT

Hi @niketn

Could you please help me out here. I have a little different scenario. We are integrating the json logs via HEC into Splunk Heavy Forwarder.

I have tried the below configurations.I am applying the props for the source.

In transforms, there are different regexes and I would want to route it to different indexes based on log files and route all the other files not required to a null queue. I would not be able to use FORMAT=indexqueue in transforms.conf as I cannot mention multiple indexes in inputs.conf .This is not working and no data is getting indexed. Kindly help.

The configs are like below:

PROPS.CONF --

[source::*model-app*]
TRANSFORMS-segment=setnull,security_logs,application_logs,provisioning_logs

TRANSFORMS.CONF --

[setnull]
REGEX=class\"\:\"(.*?)\"
DEST_KEY = queue
FORMAT = nullQueue

[security_logs]
REGEX=(class\"\:\"(/var/log/cron|/var/log/audit/audit.log|/var/log/messages|/var/log/secure)\")
DEST_KEY=_MetaData:Index
FORMAT=model_sec
WRITE_META=true
LOOKAHEAD=40000

[application_logs]
REGEX=(class\"\:\"(/var/log/application.log|/var/log/local*?.log)\")
DEST_KEY=_MetaData:Index
FORMAT=model_app
WRITE_META=true
LOOKAHEAD=40000

[provisioning_logs]
REGEX=class\"\:\"(/opt/provgw-error_msg.log|/opt/provgw-bulkrequest.log|/opt/provgw/provgw-spml_command.log.*?)\"
DEST_KEY=_MetaData:Index
FORMAT=model_prov
WRITE_META=true