topic Re: filter events based on regex and index remaining - props and transforms in Getting Data In

filter events based on regex and index remaining - props and transforms

sarnagar — Tue, 29 Sep 2020 18:17:48 GMT

Im trying to filter out events based on regex and index the remaining events based on below configs..But it doesn't seem to work...Can someone pls help..

In props.conf

[sourcetypename]
TRANSFORMS-set= setnull,setparsing

In transforms.conf

[setnull]
REGEX = (setting all transactions (.) transaction cases)|(Types:\s[.])|(FindingCall)|(Clearing junk and context)
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

Re: filter events based on regex and index remaining - props and transforms

493669 — Sun, 04 Mar 2018 08:57:23 GMT

here use [setnull] only no need of [setparsing]...remove it and then restart...and if still not working then share your sample events which you need to filter out.

Re: filter events based on regex and index remaining - props and transforms

sarnagar — Tue, 29 Sep 2020 18:18:25 GMT

Hi @493669 ,

I tried removing the setparsing part and it removes all the events and nothing is indexed.
I need only lines that contain those phrases to be removed.

Need to ignore lines containing below words:
"Preparing Call"
"Clearing Module Context"
"Types: []"
"Committing all transactions using () transaction manager"

Regex used:

(Committing all transactions using (.) transaction manager)|(Types:\s[.])|(Preparing Call)|(Clearing Module Context)

Ex of Logfile:

20180302 05:02:28,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_55808724927] - {pstm-47402055} Types: [java.lang.Long, java.lang.String, java.lang.String, java.lang.Long, null, null, java.lang.String, java.lang.String, null, java.sql.Timestamp, null, null, java.lang.Long, null, null, null]
20180302 05:02:28,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_58055999998_58072708329_7001] - {pstm-47402058} Types: [java.lang.Long]
20180302 05:02:28,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_61129156140] - {pstm-47402059} Types: [java.lang.Long]
20180302 05:02:28,858 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_61129156140] - {pstm-47402061} Types: [java.lang.Long, java.lang.String, java.lang.String, java.lang.Long, null, null, java.lang.String, java.lang.String, null, java.sql.Timestamp, null, null, java.lang.Long, null, null, null]
20180302 05:02:28,859 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_58055999998_58072708329_7001] - {pstm-47402064} Types: [java.lang.Long]
20180302 05:02:28,859 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_55808724927] - {pstm-47402065} Types: [java.lang.Long, java.lang.Long, java.lang.Long]
20180302 05:02:28,866 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_61129156140] - {pstm-47402068} Types: [java.lang.Long, java.lang.Long, java.lang.Long]
20180302 05:02:28,866 EST DEBUG sql.PreparedStatement [ConstraintManagerService_10027886_59615007125_59266246402_3003] - {pstm-47402075} Types: [java.lang.Long, null, null]
20180302 05:02:28,866 EST DEBUG sql.PreparedStatement [ConstraintManagerService_10027886_35770283302_35772759397_3003] - {pstm-47402079} Types: [java.lang.Long, null, null]
20180302 05:02:28,866 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_58055999998_58072708329_7001] - {pstm-47402081} Types: [java.lang.Long]
20180302 05:02:28,866 EST DEBUG sql.PreparedStatement [ConstraintManagerService_10027886_45166366355_45172697064_3003] - {pstm-47402074} Types: [java.lang.Long, null, null]
For NOT "Committing all transactions using (*) transaction manager"
20180302 05:04:27,239 EST INFO workflow.TransactionalWorkflowOperation [InvestablePortfolioConstraints_10027886_56399425167_56411171958] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,253 EST INFO workflow.TransactionalWorkflowOperation [OptimizerService_10027886_63883247085_62514894824_4010] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,255 EST INFO workflow.TransactionalWorkflowOperation [ClientPortfolioDatamartLoadService_10027886_55546646750_55555552651_1999] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,266 EST INFO workflow.TransactionalWorkflowOperation
Clearing Module Context asdfhasduoifhuiase\djlkfgasdui [PostOptPortfolioScores_10027886_42388259809_42393917628_8008] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,274 EST INFO workflow.TransactionalWorkflowOperation [PostOptPortfolioScores_10027886_62087947442_62515891686_8008] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,278 EST INFO workflow.TransactionalWorkflowOperation [WorkflowManager_10027886_52909311282_52946733021_4000] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,282 EST INFO workflow.TransactionalWorkflowOperation [ClientPortfolioInitializationService_10027886_35383068482_35385507500_2700] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180305 06:10:43,829 EST INFO workflow.TransactionalWorkflowOperation [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180305 06:10:43,848 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499544} Connection
20180305 06:10:43,848 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499544} Preparing Call: { call SA_TAO_AIP_PORTFOLIO_PKG.GET_ALL_INV_PORTF_FOR_CLIENT(?, ?) }
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Executing Statement: { call SA_TAO_AIP_PORTFOLIO_PKG.GET_ALL_INV_PORTF_FOR_CLIENT(?, ?) }
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Parameters: [55858397233]
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Types: [java.lang.Long]
20180305 06:10:43,856 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499546} Connection
20180305 06:10:43,856 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499546} Preparing Call: { call SA_TAO_AIP_OPT_TEMPLATE_PKG.get_portf_strtgy_ovrrde(?,?) }
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Executing Statement: { call SA_TAO_AIP_OPT_TEMPLATE_PKG.get_portf_strtgy_ovrrde(?,?) }
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Parameters: [55868440408]
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Types: [java.lang.Long]
20180305 06:10:43,867 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499548} Connection
20180305 06:10:43,867 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499548} Preparing Call: { call sa_tao_aip_portf_metric_pkg.get_portf_metric_val(?, ?, ?) }

Re: filter events based on regex and index remaining - props and transforms

493669 — Mon, 05 Mar 2018 16:37:50 GMT

just for my clarification, Ex of Logfile which you have provided contain events which you need to filter out only or you have provided all sample events.
also try this regex:

(?m)(Committing all transactions using \(.*\) transaction manager)|(Types:\s\[.*\])|(Preparing Call)|(Clearing Module Context)

Re: filter events based on regex and index remaining - props and transforms

493669 — Mon, 05 Mar 2018 17:30:04 GMT

refer this:
http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad#Filter_WMI_and_Event_Log_events

Re: filter events based on regex and index remaining - props and transforms

sarnagar — Tue, 29 Sep 2020 18:19:07 GMT

Hi @493669 ,

It still doesnt work. It nulls entire log.

Here is the complete sample log

20180305 06:10:43,769 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499536} Connection
20180305 06:10:43,769 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499536} Preparing Call: { call SA_TAO_AIP_CASH_MGT_TXN_PKG.get_cash_txn_dtl_by_cp_id(?,?) }
20180305 06:10:43,769 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499537} Executing Statement: { call SA_TAO_AIP_CASH_MGT_TXN_PKG.get_cash_txn_dtl_by_cp_id(?,?) }
20180305 06:10:43,769 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499537} Parameters: [55858397233]
20180305 06:10:43,769 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499537} Types: [java.lang.Long]
20180305 06:10:43,774 EST DEBUG sql.Connection [WorkflowManager_10027887_55656942908_55666096052_3006] - {conn-11499538} Connection
20180305 06:10:43,774 EST DEBUG sql.Connection [WorkflowManager_10027887_55656942908_55666096052_3006] - {conn-11499538} Preparing Call: { call SA_TAO_AIP_COMMON_PKG.GET_PORTF_INSTANCE_LOTS(?, ?) }
20180305 06:10:43,774 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55656942908_55666096052_3006] - {pstm-11499539} Executing Statement: { call SA_TAO_AIP_COMMON_PKG.GET_PORTF_INSTANCE_LOTS(?, ?) }
20180305 06:10:43,774 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55656942908_55666096052_3006] - {pstm-11499539} Parameters: [398207947]
20180305 06:10:43,774 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55656942908_55666096052_3006] - {pstm-11499539} Types: [java.lang.Long]
20180305 06:10:43,791 EST INFO domain.CashDomain [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - Available Cash for [investablePortfolioId=55868440408, sleeve=ROP, lot=INITIAL, withLiq=false] is: 1219.5500000
20180305 06:10:43,791 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499540} Connection
20180305 06:10:43,791 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499540} Preparing Call: { ? = call sa_tao_aip_parameter_pkg.get_ip_portf_param_value(?, ?) }
20180305 06:10:43,791 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499541} Executing Statement: { ? = call sa_tao_aip_parameter_pkg.get_ip_portf_param_value(?, ?) }
20180305 06:10:43,791 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499541} Parameters: [CASH_TXN_THRESHOLD_DAYS, 55868440408]
20180305 06:10:43,791 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499541} Types: [java.lang.String, java.lang.Long]
20180305 06:10:43,801 EST INFO domain.CashDomain [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - Cash transaction threshold days value for [investablePortfolioId=55868440408] is: 60
20180305 06:10:43,802 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499542} Connection
20180305 06:10:43,802 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499542} Preparing Call: { call sa_tao_aip_portf_metric_pkg.ins_portf_metric_val(?) }
20180305 06:10:43,802 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499543} Executing Statement: { call sa_tao_aip_portf_metric_pkg.ins_portf_metric_val(?) }
20180305 06:10:43,802 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499543} Parameters: [oracle.sql.ARRAY@10b46d2d]
20180305 06:10:43,802 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499543} Types: [oracle.sql.ARRAY]
20180305 06:10:43,829 EST INFO workflow.TransactionalWorkflowOperation [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - Returning to Workflow manager, returnCode[0]
20180305 06:10:43,829 EST INFO workflow.TransactionalWorkflowOperation [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180305 06:10:43,848 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499544} Connection
20180305 06:10:43,848 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499544} Preparing Call: { call SA_TAO_AIP_PORTFOLIO_PKG.GET_ALL_INV_PORTF_FOR_CLIENT(?, ?) }
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Executing Statement: { call SA_TAO_AIP_PORTFOLIO_PKG.GET_ALL_INV_PORTF_FOR_CLIENT(?, ?) }
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Parameters: [55858397233]
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Types: [java.lang.Long]
20180305 06:10:43,856 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499546} Connection
20180305 06:10:43,856 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499546} Preparing Call: { call SA_TAO_AIP_OPT_TEMPLATE_PKG.get_portf_strtgy_ovrrde(?,?) }
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Executing Statement: { call SA_TAO_AIP_OPT_TEMPLATE_PKG.get_portf_strtgy_ovrrde(?,?) }
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Parameters: [55868440408]
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Types: [java.lang.Long]
20180305 06:10:43,867 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499548} Connection
20180305 06:10:43,867 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499548} Preparing Call: { call sa_tao_aip_portf_metric_pkg.get_portf_metric_val(?, ?, ?) }
20180305 06:10:43,867 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499549} Executing Statement: { call sa_tao_aip_portf_metric_pkg.get_portf_metric_val(?, ?, ?) }
20180305 06:10:43,867 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499549} Parameters: [10027887, 55868440408]
20180305 06:10:43,867 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499549} Types: [java.lang.Long, java.lang.Long]

Re: filter events based on regex and index remaining - props and transforms

493669 — Tue, 06 Mar 2018 15:47:26 GMT

are you using standalone or clustered environment?
and where you have placed props.conf in Heavy forwarder or indexer?

Re: filter events based on regex and index remaining - props and transforms

sarnagar — Tue, 06 Mar 2018 16:00:26 GMT

I'm using clustered env.
I've placed props n transforms on heavy forwarder

Re: filter events based on regex and index remaining - props and transforms

493669 — Wed, 07 Mar 2018 17:57:23 GMT

ok then try this in props.conf:

[sourcetypename]
TRANSFORMS-null= setnull

in transforms.conf

[setnull]
REGEX =(?i)Committing\sall\stransactions\susing\s\(.*\)\stransaction\smanager
DEST_KEY = queue
FORMAT = nullQueue

try this and if it works then add remaining regex...

after making changes restart forwarder