topic Re: How to remove user info events from Splunk? in Splunk Search

How to remove user info events from Splunk?

dhavamanis — Mon, 28 Sep 2020 17:21:59 GMT

Can you please tell us, how to scrub remove events from Splunk indexed data (index="idx" and source="error_log"). We have indexed application server log, that contain some the event as user info details and we don't want to show those data in the splunk web-ui or keep it splunk itself. Can you please provide the step by step configuration details how to remove these events.

We want to scrub the certain pattern of event in search results. the event log contains "[error] {'username'" OR "[error] {'_updated'" pattern and no need to display in the search results. Can you please provide the configuration details.

Additional data :

Can you please provide configuration details with the below event as example how to obfuscate certain pattern of data in the event.

[Tue Aug 05 06:55:40 2014] [error] {'_updated': '2013-08-20T02:00:45.233000', 'username': 'jjjjjj1111', 'gender': 'm', '_last_login': '2011-12-07T15:03:10', 'status': 'active', 'birthdate': {'year': 1990, 'day': 1, 'month': 1}, 'address': [{'city': None, 'address1': None, 'address2': None, 'primary':True, 'state': None, 'country': None, 'postalcode': '60435', 'type': 'home'}], '_created': '2011-03-07T19:28:20', '_id':'df15fe711f964be1a2d6cb7a9b55d1234', 'email': [{'verified': False, 'primary': True, 'address': 'abcd@xyz.com'}], '_provider': {'abc':'92dd4ddb424d58b16b0c2d62908071e4'}}

[Wed Aug 20 06:50:45 2014] [error] {'username': 'sss1234', 'status': 'active', 'firstname': 'test', 'lastname': 'werq', '_last_login': '2014-08-03T03:24:17.584000', 'address': [{'city': '11111', 'address1': None, 'address2': None, 'primary': True, 'state': None, 'country': 'US', 'postalcode': '11111', 'type': 'home'}], 'brand_data': {'charcade': {'GL_UID': None, 'GL_CHALLENGEEMAILOPTOUT': None}}, '_logged_in': True, '_updated': '2014-08-03T03:24:17.614000', 'gender': 'm', 'birthdate': {'year': 2000, 'day': 1, 'month': 1}, 'avatar': 'i124.jpg', '_created': '2008-08-26T17:42:43', '_id': 'f3ddb3cd5ca14442afb8fe7dd2625c12', 'email': [{'verified': False, 'primary': True, 'address': 'qwer@xyz.com'}], '_provider': {'abc': '00f7f97140d2c3747ab7e73d55094712'}}

In the above events we want to obfuscate user identification data values like email, username and birthdate data during the indexing time.

Re: How to remove user info events from Splunk?

Richfez — Tue, 19 Aug 2014 18:44:29 GMT

Read this carefully, will it do what you need done?

http://docs.splunk.com/Documentation/Splunk/6.1.3/Indexer/RemovedatafromSplunk

Re: How to remove user info events from Splunk?

dhavamanis — Tue, 19 Aug 2014 19:00:03 GMT

Re: How to remove user info events from Splunk?

somesoni2 — Tue, 19 Aug 2014 19:24:34 GMT

I think you would have to manually delete the events you don't want. Additionally, you would like to setup ignoring those events from being indexed into splunk in future.

To Delete

Search:

index="idx" and source="error_log" "[error] {'username'" OR "[error] {'_updated'"

Ensure that it selects only the events that you don't want. Once validated, add "| delete". (read the link shared by @rich7177 for full step by step guidance on the same).

To exclude those events from being indexed itself, setup event filter for the source/sourcetype, see these:

http://answers.splunk.com/answers/1888/How-do-I-configure-Splunk-to-filter-out-events-I-don%E2%80%99t-want-to-index%3F

http://answers.splunk.com/answers/107605/filtering-events-out-via-propsconf-and-transformsconf
http://answers.splunk.com/answers/132219/filter-events-on-indexer-from-multiple-universal-forwarders

Update

Try adding this in your props.conf (on Indexer)

[YourSourceType]
SEDCMD-anonymizeData = s/'username': '(\w+)'/'username': 'XXXXXX'/g s/'address': '[\w+@\.]+'/'address': 'XXXXXX'/g s/'birthdate': \{[\w+,\.'\s:\d+]+\}/'birthdate': 'XXXXXX'/g

Re: How to remove user info events from Splunk?

dhavamanis — Wed, 20 Aug 2014 21:24:02 GMT

we want to obfuscate certain pattern of data in the event. Please refer the updated request and provide the details.

Re: How to remove user info events from Splunk?

dhavamanis — Fri, 22 Aug 2014 16:34:48 GMT

thank you so much!