Getting Data In

filter events based on regex and index remaining - props and transforms

sarnagar
Contributor

Im trying to filter out events based on regex and index the remaining events based on below configs..But it doesn't seem to work...Can someone pls help..

In props.conf

[sourcetypename]
TRANSFORMS-set= setnull,setparsing

In transforms.conf

[setnull]
REGEX = (setting all transactions (.) transaction cases)|(Types:\s[.])|(FindingCall)|(Clearing junk and context)
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

0 Karma

493669
Super Champion

here use [setnull] only no need of [setparsing]...remove it and then restart...and if still not working then share your sample events which you need to filter out.

0 Karma

sarnagar
Contributor

Hi @493669 ,

I tried removing the setparsing part and it removes all the events and nothing is indexed.
I need only lines that contain those phrases to be removed.

Need to ignore lines containing below words:
"Preparing Call"
"Clearing Module Context"
"Types: []"
"Committing all transactions using (
) transaction manager"

Regex used:

(Committing all transactions using (.) transaction manager)|(Types:\s[.])|(Preparing Call)|(Clearing Module Context)

Ex of Logfile:

20180302 05:02:28,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_55808724927] - {pstm-47402055} Types: [java.lang.Long, java.lang.String, java.lang.String, java.lang.Long, null, null, java.lang.String, java.lang.String, null, java.sql.Timestamp, null, null, java.lang.Long, null, null, null]
20180302 05:02:28,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_58055999998_58072708329_7001] - {pstm-47402058} Types: [java.lang.Long]
20180302 05:02:28,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_61129156140] - {pstm-47402059} Types: [java.lang.Long]
20180302 05:02:28,858 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_61129156140] - {pstm-47402061} Types: [java.lang.Long, java.lang.String, java.lang.String, java.lang.Long, null, null, java.lang.String, java.lang.String, null, java.sql.Timestamp, null, null, java.lang.Long, null, null, null]
20180302 05:02:28,859 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_58055999998_58072708329_7001] - {pstm-47402064} Types: [java.lang.Long]
20180302 05:02:28,859 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_55808724927] - {pstm-47402065} Types: [java.lang.Long, java.lang.Long, java.lang.Long]
20180302 05:02:28,866 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_61129156140] - {pstm-47402068} Types: [java.lang.Long, java.lang.Long, java.lang.Long]
20180302 05:02:28,866 EST DEBUG sql.PreparedStatement [ConstraintManagerService_10027886_59615007125_59266246402_3003] - {pstm-47402075} Types: [java.lang.Long, null, null]
20180302 05:02:28,866 EST DEBUG sql.PreparedStatement [ConstraintManagerService_10027886_35770283302_35772759397_3003] - {pstm-47402079} Types: [java.lang.Long, null, null]
20180302 05:02:28,866 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_58055999998_58072708329_7001] - {pstm-47402081} Types: [java.lang.Long]
20180302 05:02:28,866 EST DEBUG sql.PreparedStatement [ConstraintManagerService_10027886_45166366355_45172697064_3003] - {pstm-47402074} Types: [java.lang.Long, null, null]
For NOT "Committing all transactions using (*) transaction manager"
20180302 05:04:27,239 EST INFO workflow.TransactionalWorkflowOperation [InvestablePortfolioConstraints_10027886_56399425167_56411171958] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,253 EST INFO workflow.TransactionalWorkflowOperation [OptimizerService_10027886_63883247085_62514894824_4010] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,255 EST INFO workflow.TransactionalWorkflowOperation [ClientPortfolioDatamartLoadService_10027886_55546646750_55555552651_1999] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,266 EST INFO workflow.TransactionalWorkflowOperation
Clearing Module Context asdfhasduoifhuiase\djlkfgasdui [PostOptPortfolioScores_10027886_42388259809_42393917628_8008] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,274 EST INFO workflow.TransactionalWorkflowOperation [PostOptPortfolioScores_10027886_62087947442_62515891686_8008] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,278 EST INFO workflow.TransactionalWorkflowOperation [WorkflowManager_10027886_52909311282_52946733021_4000] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,282 EST INFO workflow.TransactionalWorkflowOperation [ClientPortfolioInitializationService_10027886_35383068482_35385507500_2700] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180305 06:10:43,829 EST INFO workflow.TransactionalWorkflowOperation [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180305 06:10:43,848 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499544} Connection
20180305 06:10:43,848 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499544} Preparing Call: { call SA_TAO_AIP_PORTFOLIO_PKG.GET_ALL_INV_PORTF_FOR_CLIENT(?, ?) }
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Executing Statement: { call SA_TAO_AIP_PORTFOLIO_PKG.GET_ALL_INV_PORTF_FOR_CLIENT(?, ?) }
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Parameters: [55858397233]
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Types: [java.lang.Long]
20180305 06:10:43,856 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499546} Connection
20180305 06:10:43,856 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499546} Preparing Call: { call SA_TAO_AIP_OPT_TEMPLATE_PKG.get_portf_strtgy_ovrrde(?,?) }
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Executing Statement: { call SA_TAO_AIP_OPT_TEMPLATE_PKG.get_portf_strtgy_ovrrde(?,?) }
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Parameters: [55868440408]
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Types: [java.lang.Long]
20180305 06:10:43,867 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499548} Connection
20180305 06:10:43,867 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499548} Preparing Call: { call sa_tao_aip_portf_metric_pkg.get_portf_metric_val(?, ?, ?) }

0 Karma

493669
Super Champion

just for my clarification, Ex of Logfile which you have provided contain events which you need to filter out only or you have provided all sample events.
also try this regex:

(?m)(Committing all transactions using \(.*\) transaction manager)|(Types:\s\[.*\])|(Preparing Call)|(Clearing Module Context)
0 Karma

493669
Super Champion
0 Karma

sarnagar
Contributor

Hi @493669 ,

It still doesnt work. It nulls entire log.

Here is the complete sample log

20180305 06:10:43,769 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499536} Connection
20180305 06:10:43,769 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499536} Preparing Call: { call SA_TAO_AIP_CASH_MGT_TXN_PKG.get_cash_txn_dtl_by_cp_id(?,?) }
20180305 06:10:43,769 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499537} Executing Statement: { call SA_TAO_AIP_CASH_MGT_TXN_PKG.get_cash_txn_dtl_by_cp_id(?,?) }
20180305 06:10:43,769 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499537} Parameters: [55858397233]
20180305 06:10:43,769 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499537} Types: [java.lang.Long]
20180305 06:10:43,774 EST DEBUG sql.Connection [WorkflowManager_10027887_55656942908_55666096052_3006] - {conn-11499538} Connection
20180305 06:10:43,774 EST DEBUG sql.Connection [WorkflowManager_10027887_55656942908_55666096052_3006] - {conn-11499538} Preparing Call: { call SA_TAO_AIP_COMMON_PKG.GET_PORTF_INSTANCE_LOTS(?, ?) }
20180305 06:10:43,774 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55656942908_55666096052_3006] - {pstm-11499539} Executing Statement: { call SA_TAO_AIP_COMMON_PKG.GET_PORTF_INSTANCE_LOTS(?, ?) }
20180305 06:10:43,774 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55656942908_55666096052_3006] - {pstm-11499539} Parameters: [398207947]
20180305 06:10:43,774 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55656942908_55666096052_3006] - {pstm-11499539} Types: [java.lang.Long]
20180305 06:10:43,791 EST INFO domain.CashDomain [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - Available Cash for [investablePortfolioId=55868440408, sleeve=ROP, lot=INITIAL, withLiq=false] is: 1219.5500000
20180305 06:10:43,791 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499540} Connection
20180305 06:10:43,791 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499540} Preparing Call: { ? = call sa_tao_aip_parameter_pkg.get_ip_portf_param_value(?, ?) }
20180305 06:10:43,791 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499541} Executing Statement: { ? = call sa_tao_aip_parameter_pkg.get_ip_portf_param_value(?, ?) }
20180305 06:10:43,791 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499541} Parameters: [CASH_TXN_THRESHOLD_DAYS, 55868440408]
20180305 06:10:43,791 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499541} Types: [java.lang.String, java.lang.Long]
20180305 06:10:43,801 EST INFO domain.CashDomain [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - Cash transaction threshold days value for [investablePortfolioId=55868440408] is: 60
20180305 06:10:43,802 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499542} Connection
20180305 06:10:43,802 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499542} Preparing Call: { call sa_tao_aip_portf_metric_pkg.ins_portf_metric_val(?) }
20180305 06:10:43,802 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499543} Executing Statement: { call sa_tao_aip_portf_metric_pkg.ins_portf_metric_val(?) }
20180305 06:10:43,802 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499543} Parameters: [oracle.sql.ARRAY@10b46d2d]
20180305 06:10:43,802 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499543} Types: [oracle.sql.ARRAY]
20180305 06:10:43,829 EST INFO workflow.TransactionalWorkflowOperation [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - Returning to Workflow manager, returnCode[0]
20180305 06:10:43,829 EST INFO workflow.TransactionalWorkflowOperation [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180305 06:10:43,848 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499544} Connection
20180305 06:10:43,848 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499544} Preparing Call: { call SA_TAO_AIP_PORTFOLIO_PKG.GET_ALL_INV_PORTF_FOR_CLIENT(?, ?) }
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Executing Statement: { call SA_TAO_AIP_PORTFOLIO_PKG.GET_ALL_INV_PORTF_FOR_CLIENT(?, ?) }
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Parameters: [55858397233]
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Types: [java.lang.Long]
20180305 06:10:43,856 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499546} Connection
20180305 06:10:43,856 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499546} Preparing Call: { call SA_TAO_AIP_OPT_TEMPLATE_PKG.get_portf_strtgy_ovrrde(?,?) }
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Executing Statement: { call SA_TAO_AIP_OPT_TEMPLATE_PKG.get_portf_strtgy_ovrrde(?,?) }
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Parameters: [55868440408]
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Types: [java.lang.Long]
20180305 06:10:43,867 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499548} Connection
20180305 06:10:43,867 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499548} Preparing Call: { call sa_tao_aip_portf_metric_pkg.get_portf_metric_val(?, ?, ?) }
20180305 06:10:43,867 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499549} Executing Statement: { call sa_tao_aip_portf_metric_pkg.get_portf_metric_val(?, ?, ?) }
20180305 06:10:43,867 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499549} Parameters: [10027887, 55868440408]
20180305 06:10:43,867 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499549} Types: [java.lang.Long, java.lang.Long]

0 Karma

493669
Super Champion

are you using standalone or clustered environment?
and where you have placed props.conf in Heavy forwarder or indexer?

0 Karma

sarnagar
Contributor

I'm using clustered env.
I've placed props n transforms on heavy forwarder

0 Karma

493669
Super Champion

ok then try this in props.conf:

[sourcetypename]
TRANSFORMS-null= setnull

in transforms.conf

[setnull]
REGEX =(?i)Committing\sall\stransactions\susing\s\(.*\)\stransaction\smanager
DEST_KEY = queue
FORMAT = nullQueue

try this and if it works then add remaining regex...

after making changes restart forwarder

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!