How to remove user info events from Splunk?

[dhavamanis]

Can you please tell us, how to scrub remove events from Splunk indexed data (index="idx" and source="error_log"). We have indexed application server log, that contain some the event as user info details and we don't want to show those data in the splunk web-ui or keep it splunk itself. Can you please provide the step by step configuration details how to remove these events.

We want to scrub the certain pattern of event in search results. the event log contains "[error] {'username'" OR "[error] {'_updated'" pattern and no need to display in the search results. Can you please provide the configuration details.

Additional data :

Can you please provide configuration details with the below event as example how to obfuscate certain pattern of data in the event.

[Tue Aug 05 06:55:40 2014] [error] {'_updated': '2013-08-20T02:00:45.233000', 'username': 'jjjjjj1111', 'gender': 'm', '_last_login': '2011-12-07T15:03:10', 'status': 'active', 'birthdate': {'year': 1990, 'day': 1, 'month': 1}, 'address': [{'city': None, 'address1': None, 'address2': None, 'primary':True, 'state': None, 'country': None, 'postalcode': '60435', 'type': 'home'}], '_created': '2011-03-07T19:28:20', '_id':'df15fe711f964be1a2d6cb7a9b55d1234', 'email': [{'verified': False, 'primary': True, 'address': 'abcd@xyz.com'}], '_provider': {'abc':'92dd4ddb424d58b16b0c2d62908071e4'}}

[Wed Aug 20 06:50:45 2014] [error] {'username': 'sss1234', 'status': 'active', 'firstname': 'test', 'lastname': 'werq', '_last_login': '2014-08-03T03:24:17.584000', 'address': [{'city': '11111', 'address1': None, 'address2': None, 'primary': True, 'state': None, 'country': 'US', 'postalcode': '11111', 'type': 'home'}], 'brand_data': {'charcade': {'GL_UID': None, 'GL_CHALLENGEEMAILOPTOUT': None}}, '_logged_in': True, '_updated': '2014-08-03T03:24:17.614000', 'gender': 'm', 'birthdate': {'year': 2000, 'day': 1, 'month': 1}, 'avatar': 'i124.jpg', '_created': '2008-08-26T17:42:43', '_id': 'f3ddb3cd5ca14442afb8fe7dd2625c12', 'email': [{'verified': False, 'primary': True, 'address': 'qwer@xyz.com'}], '_provider': {'abc': '00f7f97140d2c3747ab7e73d55094712'}}

In the above events we want to obfuscate user identification data values like email, username and birthdate data during the indexing time.

[somesoni2 · Accepted solution]

I think you would have to manually delete the events you don't want. Additionally, you would like to setup ignoring those events from being indexed into splunk in future.

To Delete

Search:

index="idx" and source="error_log" "[error] {'username'" OR "[error] {'_updated'"

Ensure that it selects only the events that you don't want. Once validated, add "| delete". (read the link shared by @rich7177 for full step by step guidance on the same).

To exclude those events from being indexed itself, setup event filter for the source/sourcetype, see these:

http://answers.splunk.com/answers/1888/How-do-I-configure-Splunk-to-filter-out-events-I-don%E2%80%99...

http://answers.splunk.com/answers/107605/filtering-events-out-via-propsconf-and-transformsconf
http://answers.splunk.com/answers/132219/filter-events-on-indexer-from-multiple-universal-forwarders

Update

Try adding this in your props.conf (on Indexer)

[YourSourceType]
SEDCMD-anonymizeData = s/'username': '(\w+)'/'username': 'XXXXXX'/g s/'address': '[\w+@\.]+'/'address': 'XXXXXX'/g s/'birthdate': \{[\w+,\.'\s:\d+]+\}/'birthdate': 'XXXXXX'/g

[dhavamanis]

thank you so much!

[dhavamanis]

we want to obfuscate certain pattern of data in the event. Please refer the updated request and provide the details.

[Richfez]

Read this carefully, will it do what you need done?

http://docs.splunk.com/Documentation/Splunk/6.1.3/Indexer/RemovedatafromSplunk

[dhavamanis]

We want to scrub the certain pattern of event in search results. the event log contains "[error] {'username'" OR "[error] {'_updated'" pattern and no need to display in the search results. Can you please provide the configuration details.