topic Re: How to remove a specific string from an events in splunk ? in Getting Data In

How to remove a specific string from an events in splunk ?

Hemnaath — Tue, 29 Sep 2020 14:05:28 GMT

Hi All, currently we are facing an issue in removing a specific values from the event list starting with the word "at" as we do not want these in the splunk events.

Example :

5/16/17
8:57:04.674 AM

[2017-05-16T08:57:04.674-04:00] [TIM_server1] [ERROR] [] [db2.tam.platform.kernel.impl] [tid: [ACTIVE].ExecuteThread: '29' for queue: 'xxxx.kernel.Default (self-tuning)'] [userId: xxx_TIMPASSSYNC] [ecid: d87cb14ef99e9513:-133033ab:15c0857bfd3:-8000-000000000005cb77,0] [APP: TIM#11.1.2.0.0] [J2EE_APP.name: spml-dsml] [J2EE_MODULE.name: spmlws] [WEBSERVICE.name: TIMProvisioning] [WEBSERVICE_PORT.name: TIMProvisioningPort] Kernel Information: {0}[[
db2.tam.platform.kernel.ValidationFailedException: IAM-3030006:The following password policy rules were not met:Password must not be one of 8 previous passwords.
:
at db2.tam.passwordmgmt.eventhandlers.UserPasswordValidationHandler.validate(UserPasswordValidationHandler.java:96)
at db2.tam.platform.kernel.impl.TIMEvent.executeHandlers(TIMEvent.java:204)
at db2.tam.platform.kernel.impl.MonitoredTIMEvent.invokeExecuteHandler(MonitoredTIMEvent.java:99)
at db2.tam.platform.kernel.impl.MonitoredTIMEvent.executeHandlers(MonitoredTIMEvent.java:69)
at db2.tam.platform.kernel.impl.TIMEvent.execute(TIMEvent.java:157)
at db2.tam.platform.kernel.impl.ProcessImpl.executeStage(ProcessImpl.java:223)
at db2.tam.platform.kernel.impl.TIMProcess.doStageExecution(TIMProcess.java:38)
at db2.tam.platform.kernel.impl.ProcessImpl.execute(ProcessImpl.java:182)
at db2.tam.platform.kernel.impl.MonitoredTIMProcess.execute(MonitoredTIMProcess.java:33)
at db2.tam.platform.kernel.impl.Utils.manageSyncProcessing(Utils.java:73)

Kindly guide me on how to remove this value starting with "at" from the events list.

thanks in advance

Re: How to remove a specific string from an events in splunk ?

somesoni2 — Tue, 16 May 2017 17:10:55 GMT

Try this

props.conf on indexer/heavy forwarder

[yoursourcetypehere]
..other configs..
SEDCMD-removeat = s/at \S+//g

Re: How to remove a specific string from an events in splunk ?

Hemnaath — Tue, 16 May 2017 17:26:04 GMT

Hi Somesoni2 thanks for your effort, actually we are monitoring the identity management logs in splunk and we are using UF agent to get the events from remote machine to the splunk indexer. so in this case were do we want to configure the props.conf details in indexer or Heavy forwarder. And also do we need to configure transforms.conf along with props.conf

Props.conf : Is the below stanza correct
[sourcetype=IBM:AUT:TAM]
SEDCMD-removeat = s/at /s+//g

kindly guide me on this.
thanks in advance

Re: How to remove a specific string from an events in splunk ?

Hemnaath — Tue, 16 May 2017 17:52:37 GMT

Somesoni2 kindly guide me on how to remove a specific string from an event in splunk.

thanks in advance.

Re: How to remove a specific string from an events in splunk ?

somesoni2 — Tue, 16 May 2017 17:53:04 GMT

A UF is not heavy forwarder (Splunk Enterprise instance) so the configurations should go to Indexers. There are two methods, one involves transforms.conf and one (this one) doesn't so you don't need a transforms.conf with this method.

This should be your configuration in props.conf (no prefix for sourcetype and it's upper case S in the regex).

[IBM:AUT:TAM]
SEDCMD-removeat = s/at /S+//g

Re: How to remove a specific string from an events in splunk ?

Hemnaath — Tue, 29 Sep 2020 14:11:34 GMT

Hi Somesoni2, I have tried the above stanza inprod environment but it is not working, we could still see the huge log details getting into splunk.

Props.conf details: Placed at Indexer instances.

[ibm:auth:identitymanagement]
SEDCMD-removeat = s/at /S+//g

Log details :

[2017-05-24T09:21:31.473-04:00] [itm_server1] [ERROR] [] [com.xxxxx.tam.itm.plugins.eventhandlers] [tid: [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: xxxxx] [ecid: ce05a10ae6311cf9:15640c05:15c2c928226:-8000-00000000000b19be,1:123289:21] [APP: itm#11.1.2.0.0] [J2EE_APP.name: itm_11.1.2.0.0] [J2EE_MODULE.name: workflowservice] [WEBSERVICE.name: CallbackService] [WEBSERVICE_PORT.name: CallbackServicePort] Exception Occurred.[[
ibm.tam.identity.exception.AccessDeniedException: tam-3054101:The logged-in user itminternal does not have viewSearchEntity permission on Role xxxxx Inactive itm
Users entity.:itminternal:viewSearchEntity:Role:xxxxx Inactive itm Users
at ibm.tam.identity.rolemgmt.impl.RoleManagerCommon.hasAccess(RoleManagerCommon.java:401)
at ibm.tam.identity.rolemgmt.impl.RoleManagerCommon.hasAccess(RoleManagerCommon.java:251)
at ibm.tam.identity.rolemgmt.impl.RoleManagerImpl.getDetails(RoleManagerImpl.java:531)
at ibm.tam.identity.rolemgmt.impl.RoleManagerImpl.getDetails(RoleManagerImpl.java:492)
at sun.reflect.GeneratedMethodAccessor4906.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)

kindly guide me how to remove the word starting with "at" from the events list from above actual log details.

Re: How to remove a specific string from an events in splunk ?

somesoni2 — Wed, 24 May 2017 14:53:38 GMT

You're using foward slash before S+ where it should be backward slash, like in my answer.

Re: How to remove a specific string from an events in splunk ?

Hemnaath — Wed, 24 May 2017 15:18:10 GMT

thanks somesoni2, can I update the below stanza will it fix the issue.

[ibm:auth:identitymanagement]
SEDCMD-removeat = s/at \S+//g

Re: How to remove a specific string from an events in splunk ?

somesoni2 — Wed, 24 May 2017 16:20:15 GMT

It will not correct the data that is already ingested. It will do it for any new data that will come.

To validate where it works or not, (assuming you've some data that was ingested with those lines NOT removed), try like this

base search to select current full data 
| rex mode=sed "s/at \S+//g"

The search result should've those lines removed.

Re: How to remove a specific string from an events in splunk ?

Hemnaath — Tue, 29 Sep 2020 14:14:39 GMT

hey somesoni2, it worked but at the same time we could see some space between the events now and the word "at" is removed. Now the events look like this

log details :
[2017-05-24T12:40:55.515-04:00] [ibm_server1] [ERROR] [] [ibm.iam.platform.kernel.impl] [tid: [ACTIVE].ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: xxxxxxx] [ecid: 061bf930d1319f9b:-30072ed4:15c2c8bbbd3:-8000-000000000009f634,0] [APP: ibm#11.1.2.0.0] [J2EE_APP.name: spml-dsml] [J2EE_MODULE.name: spmlws] [WEBSERVICE.name: ibmProvisioning] [WEBSERVICE_PORT.name: ibmProvisioningPort] Kernel Information: {0}[[
ibm.tam.platform.kernel.ValidationFailedException: IAM-3030006:The following password policy rules were not met:Password must not be one of 8 previous passwords.
:

 Source)


 Source)

 Source)
 Source)




     Method)

Caused By: ibm.tam.passwordmgmt.exception.InvalidPasswordException

kindly guide me on how to remove the space and the words from the events.

Re: How to remove a specific string from an events in splunk ?

somesoni2 — Wed, 24 May 2017 17:49:19 GMT

Try this

 base search to select current full data 
 | rex mode=sed "s/at .+//g"

Re: How to remove a specific string from an events in splunk ?

Hemnaath — Tue, 29 Sep 2020 14:14:45 GMT

somesoni2, can I update above regx to the stanza in props.conf, as the user wants the data not to be injected into splunk. User wants only the events not the content shown in " show all 124 lines details " so in this case can I update the props stanza with above regex mentioned for a base search

Props.conf details :
[ibm:auth:identitymanagement]
SEDCMD-removeat = s/at \s .+//g

Log details :
5/24/17
12:40:55.515 PM
[2017-05-24T12:40:55.515-04:00] [itm_server1] [ERROR] [] [ibm.tam.platform.kernel.impl] [tid: [ACTIVE].ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: xxxxxx] [ecid: 061bf930d1319f9b:-30072ed4:15c2c8bbbd3:-8000-000000000009f634,0] [APP: itm#11.1.2.0.0] [J2EE_APP.name: spml-dsml] [J2EE_MODULE.name: spmlws] [WEBSERVICE.name: itmProvisioning] [WEBSERVICE_PORT.name: itmProvisioningPort] Kernel Information: {0}[[
ibm.tam.platform.kernel.ValidationFailedException: tam-3030006:The following password policy rules were not met:Password must not be one of 8 previous passwords.
:

Show all 124 lines

Kindly guide me on this as user needs only the events content not the data which are under "show all 124 lines details"

thanks in advance

Re: How to remove a specific string from an events in splunk ?

somesoni2 — Wed, 24 May 2017 18:31:20 GMT

Above regex will remove everything after the at (removed text won't be ingested). so yes, you can put above regex into props.conf (on indexer/heavy forwarder).

Re: How to remove a specific string from an events in splunk ?

Hemnaath — Wed, 24 May 2017 18:44:47 GMT

somesoni2 thanks for your effort on this, could please correct me if the below stanza can be updated in props.conf in indexer instances.

Props.conf details :
[ibm:auth:identitymanagement]
SEDCMD-removeat = s/at \S .+//g

thanks in advance.

Re: How to remove a specific string from an events in splunk ?

somesoni2 — Wed, 24 May 2017 18:53:46 GMT

Use this (exacly) s/at .+//g

Re: How to remove a specific string from an events in splunk ?

Hemnaath — Wed, 24 May 2017 18:59:03 GMT

sure, thanks. I will update the below stanza in props,conf and validate it .

Props.conf details :
[ibm:auth:identitymanagement]
SEDCMD-removeat =s/at .+//g

Re: How to remove a specific string from an events in splunk ?

Hemnaath — Wed, 24 May 2017 19:53:20 GMT

somesoni2 , we could see some of the events are still showing the blank space after updating the below props stanza in indexer instances.

Props.conf details :
[ibm:auth:identitymanagement]
SEDCMD-removeat =s/at .+//g

Log details:

5/24/17
3:39:30.000 PM

WatchData: DATE = May 24, 2017 3:39:30 PM EDT SERVER = itm_server1 MESSAGE = [ServletContext@1479415291[app:ibm.iam.console.identity.self-service.ear module:identity path:/identity spec-version:2.5 version:V2.0]] Root cause of ServletException.
java.lang.AssertionError: Assertion violated

SUBSYSTEM = HTTP USERID = SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-101017 MACHINE = hit01.xxxxx.com TXID = CONTEXTID = 061bf930d1319f9b:-30072ed4:15c2c8bbbd3:-8000-00000000000a715f TIMESTAMP = 1495654770087

WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 30000

Kindly help me to remove the space from the events t which are under "show all lines details.

Re: How to remove a specific string from an events in splunk ?

Hemnaath — Tue, 29 Sep 2020 14:15:03 GMT

Hi Somesoni2, thanks for your much need help on removing the word at and the white space from the event content, though the above regex helped to remove most of the events contained the letter at and the white space but still we could see some of the events are having the white space under the "show all lines details" as shown on below event data.

Log details:
5/25/17
2:10:00.487 AM

[2017-05-25T02:10:00.487-04:00] [itm_server1] [ERROR] [] [XELLERATE.AUDITOR] [tid: [ACTIVE].ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: itminternal] [ecid: 0000Lkg^grQ3JBc5TjO5yc1P8V12000002,0] [APP: itm#11.1.2.0.0] Processor (com.thortech.xl.audit.auditdataprocessors.UserProfileRDGenerator) failed to process successfully generated audit data[[
com.thortech.xl.audit.exceptions.AuditDataProcessingFailedException: Failed to insert form profile record in table UPA_UD_FORMS

Caused by: com.thortech.xl.orb.dataaccess.tcDataAccessException

... 39 more

Caused by: java.sql.SQLIntegrityConstraintViolationException: ORA-02291: integrity constraint (PRD_itm.FK_UPA_UD_UPA_RES) violated - parent key not found

... 41 more

]]
Collapse

kindly guide me on removing the space from events under the "show all lines details"

thanks in advance.

Re: How to remove a specific string from an events in splunk ?

somesoni2 — Thu, 25 May 2017 06:20:14 GMT

Can you paste both original event (full) and what Splunk is showing? May be a screenshot?

Re: How to remove a specific string from an events in splunk ?

somesoni2 — Thu, 25 May 2017 06:32:36 GMT

Try adding additional SEDCMD for spaces. LIke this

[ibm:auth:identitymanagement]
SEDCMD-removeat =s/at .+//g
SEDCMD-removespaces = s/\s+\s+[\r\n]//g