Getting Data In
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question

filter events based on regex and index remaining - props and transforms

sarnagar
Contributor
‎03-04-2018 12:26 AM

Im trying to filter out events based on regex and index the remaining events based on below configs..But it doesn't seem to work...Can someone pls help..

In props.conf

[sourcetypename]
TRANSFORMS-set= setnull,setparsing

In transforms.conf

[setnull]
REGEX = (setting all transactions (.) transaction cases)|(Types:\s[.])|(FindingCall)|(Clearing junk and context)
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

0 Karma
Reply

493669
Super Champion
‎03-04-2018 12:57 AM

here use [setnull] only no need of [setparsing]...remove it and then restart...and if still not working then share your sample events which you need to filter out.

0 Karma
Reply

sarnagar
Contributor
‎03-05-2018 05:25 AM

Hi @493669 ,

I tried removing the setparsing part and it removes all the events and nothing is indexed.
I need only lines that contain those phrases to be removed.

Need to ignore lines containing below words:
"Preparing Call"
"Clearing Module Context"
"Types: []"
"Committing all transactions using () transaction manager"

Regex used:

(Committing all transactions using (.) transaction manager)|(Types:\s[.])|(Preparing Call)|(Clearing Module Context)

Ex of Logfile:

20180302 05:02:28,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_55808724927] - {pstm-47402055} Types: [java.lang.Long, java.lang.String, java.lang.String, java.lang.Long, null, null, java.lang.String, java.lang.String, null, java.sql.Timestamp, null, null, java.lang.Long, null, null, null]
20180302 05:02:28,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_58055999998_58072708329_7001] - {pstm-47402058} Types: [java.lang.Long]
20180302 05:02:28,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_61129156140] - {pstm-47402059} Types: [java.lang.Long]
20180302 05:02:28,858 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_61129156140] - {pstm-47402061} Types: [java.lang.Long, java.lang.String, java.lang.String, java.lang.Long, null, null, java.lang.String, java.lang.String, null, java.sql.Timestamp, null, null, java.lang.Long, null, null, null]
20180302 05:02:28,859 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_58055999998_58072708329_7001] - {pstm-47402064} Types: [java.lang.Long]
20180302 05:02:28,859 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_55808724927] - {pstm-47402065} Types: [java.lang.Long, java.lang.Long, java.lang.Long]
20180302 05:02:28,866 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_61129156140] - {pstm-47402068} Types: [java.lang.Long, java.lang.Long, java.lang.Long]
20180302 05:02:28,866 EST DEBUG sql.PreparedStatement [ConstraintManagerService_10027886_59615007125_59266246402_3003] - {pstm-47402075} Types: [java.lang.Long, null, null]
20180302 05:02:28,866 EST DEBUG sql.PreparedStatement [ConstraintManagerService_10027886_35770283302_35772759397_3003] - {pstm-47402079} Types: [java.lang.Long, null, null]
20180302 05:02:28,866 EST DEBUG sql.PreparedStatement [WorkflowManager_10027886_58055999998_58072708329_7001] - {pstm-47402081} Types: [java.lang.Long]
20180302 05:02:28,866 EST DEBUG sql.PreparedStatement [ConstraintManagerService_10027886_45166366355_45172697064_3003] - {pstm-47402074} Types: [java.lang.Long, null, null]
For NOT "Committing all transactions using (*) transaction manager"
20180302 05:04:27,239 EST INFO workflow.TransactionalWorkflowOperation [InvestablePortfolioConstraints_10027886_56399425167_56411171958] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,253 EST INFO workflow.TransactionalWorkflowOperation [OptimizerService_10027886_63883247085_62514894824_4010] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,255 EST INFO workflow.TransactionalWorkflowOperation [ClientPortfolioDatamartLoadService_10027886_55546646750_55555552651_1999] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,266 EST INFO workflow.TransactionalWorkflowOperation
Clearing Module Context asdfhasduoifhuiase\djlkfgasdui [PostOptPortfolioScores_10027886_42388259809_42393917628_8008] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,274 EST INFO workflow.TransactionalWorkflowOperation [PostOptPortfolioScores_10027886_62087947442_62515891686_8008] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,278 EST INFO workflow.TransactionalWorkflowOperation [WorkflowManager_10027886_52909311282_52946733021_4000] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180302 05:04:27,282 EST INFO workflow.TransactionalWorkflowOperation [ClientPortfolioInitializationService_10027886_35383068482_35385507500_2700] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180305 06:10:43,829 EST INFO workflow.TransactionalWorkflowOperation [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180305 06:10:43,848 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499544} Connection
20180305 06:10:43,848 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499544} Preparing Call: { call SA_TAO_AIP_PORTFOLIO_PKG.GET_ALL_INV_PORTF_FOR_CLIENT(?, ?) }
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Executing Statement: { call SA_TAO_AIP_PORTFOLIO_PKG.GET_ALL_INV_PORTF_FOR_CLIENT(?, ?) }
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Parameters: [55858397233]
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Types: [java.lang.Long]
20180305 06:10:43,856 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499546} Connection
20180305 06:10:43,856 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499546} Preparing Call: { call SA_TAO_AIP_OPT_TEMPLATE_PKG.get_portf_strtgy_ovrrde(?,?) }
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Executing Statement: { call SA_TAO_AIP_OPT_TEMPLATE_PKG.get_portf_strtgy_ovrrde(?,?) }
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Parameters: [55868440408]
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Types: [java.lang.Long]
20180305 06:10:43,867 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499548} Connection
20180305 06:10:43,867 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499548} Preparing Call: { call sa_tao_aip_portf_metric_pkg.get_portf_metric_val(?, ?, ?) }

0 Karma
Reply

493669
Super Champion
‎03-05-2018 08:37 AM

just for my clarification, Ex of Logfile which you have provided contain events which you need to filter out only or you have provided all sample events.
also try this regex:

(?m)(Committing all transactions using \(.*\) transaction manager)|(Types:\s\[.*\])|(Preparing Call)|(Clearing Module Context)
0 Karma
Reply

493669
Super Champion
‎03-05-2018 09:30 AM

refer this:
http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad#Filter_WMI_and_Even...

0 Karma
Reply

sarnagar
Contributor
‎03-06-2018 04:56 AM

Hi @493669 ,

It still doesnt work. It nulls entire log.

Here is the complete sample log

20180305 06:10:43,769 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499536} Connection
20180305 06:10:43,769 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499536} Preparing Call: { call SA_TAO_AIP_CASH_MGT_TXN_PKG.get_cash_txn_dtl_by_cp_id(?,?) }
20180305 06:10:43,769 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499537} Executing Statement: { call SA_TAO_AIP_CASH_MGT_TXN_PKG.get_cash_txn_dtl_by_cp_id(?,?) }
20180305 06:10:43,769 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499537} Parameters: [55858397233]
20180305 06:10:43,769 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499537} Types: [java.lang.Long]
20180305 06:10:43,774 EST DEBUG sql.Connection [WorkflowManager_10027887_55656942908_55666096052_3006] - {conn-11499538} Connection
20180305 06:10:43,774 EST DEBUG sql.Connection [WorkflowManager_10027887_55656942908_55666096052_3006] - {conn-11499538} Preparing Call: { call SA_TAO_AIP_COMMON_PKG.GET_PORTF_INSTANCE_LOTS(?, ?) }
20180305 06:10:43,774 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55656942908_55666096052_3006] - {pstm-11499539} Executing Statement: { call SA_TAO_AIP_COMMON_PKG.GET_PORTF_INSTANCE_LOTS(?, ?) }
20180305 06:10:43,774 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55656942908_55666096052_3006] - {pstm-11499539} Parameters: [398207947]
20180305 06:10:43,774 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55656942908_55666096052_3006] - {pstm-11499539} Types: [java.lang.Long]
20180305 06:10:43,791 EST INFO domain.CashDomain [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - Available Cash for [investablePortfolioId=55868440408, sleeve=ROP, lot=INITIAL, withLiq=false] is: 1219.5500000
20180305 06:10:43,791 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499540} Connection
20180305 06:10:43,791 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499540} Preparing Call: { ? = call sa_tao_aip_parameter_pkg.get_ip_portf_param_value(?, ?) }
20180305 06:10:43,791 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499541} Executing Statement: { ? = call sa_tao_aip_parameter_pkg.get_ip_portf_param_value(?, ?) }
20180305 06:10:43,791 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499541} Parameters: [CASH_TXN_THRESHOLD_DAYS, 55868440408]
20180305 06:10:43,791 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499541} Types: [java.lang.String, java.lang.Long]
20180305 06:10:43,801 EST INFO domain.CashDomain [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - Cash transaction threshold days value for [investablePortfolioId=55868440408] is: 60
20180305 06:10:43,802 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499542} Connection
20180305 06:10:43,802 EST DEBUG sql.Connection [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {conn-11499542} Preparing Call: { call sa_tao_aip_portf_metric_pkg.ins_portf_metric_val(?) }
20180305 06:10:43,802 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499543} Executing Statement: { call sa_tao_aip_portf_metric_pkg.ins_portf_metric_val(?) }
20180305 06:10:43,802 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499543} Parameters: [oracle.sql.ARRAY@10b46d2d]
20180305 06:10:43,802 EST DEBUG sql.PreparedStatement [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - {pstm-11499543} Types: [oracle.sql.ARRAY]
20180305 06:10:43,829 EST INFO workflow.TransactionalWorkflowOperation [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - Returning to Workflow manager, returnCode[0]
20180305 06:10:43,829 EST INFO workflow.TransactionalWorkflowOperation [PreOptPortfolioScores_10027887_55858397233_55868440408_8001] - ---------- Committing all transactions using (workflowTransaction) transaction manager ----------
20180305 06:10:43,848 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499544} Connection
20180305 06:10:43,848 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499544} Preparing Call: { call SA_TAO_AIP_PORTFOLIO_PKG.GET_ALL_INV_PORTF_FOR_CLIENT(?, ?) }
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Executing Statement: { call SA_TAO_AIP_PORTFOLIO_PKG.GET_ALL_INV_PORTF_FOR_CLIENT(?, ?) }
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Parameters: [55858397233]
20180305 06:10:43,848 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499545} Types: [java.lang.Long]
20180305 06:10:43,856 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499546} Connection
20180305 06:10:43,856 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499546} Preparing Call: { call SA_TAO_AIP_OPT_TEMPLATE_PKG.get_portf_strtgy_ovrrde(?,?) }
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Executing Statement: { call SA_TAO_AIP_OPT_TEMPLATE_PKG.get_portf_strtgy_ovrrde(?,?) }
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Parameters: [55868440408]
20180305 06:10:43,856 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499547} Types: [java.lang.Long]
20180305 06:10:43,867 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499548} Connection
20180305 06:10:43,867 EST DEBUG sql.Connection [WorkflowManager_10027887_55858397233_55868440408_2000] - {conn-11499548} Preparing Call: { call sa_tao_aip_portf_metric_pkg.get_portf_metric_val(?, ?, ?) }
20180305 06:10:43,867 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499549} Executing Statement: { call sa_tao_aip_portf_metric_pkg.get_portf_metric_val(?, ?, ?) }
20180305 06:10:43,867 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499549} Parameters: [10027887, 55868440408]
20180305 06:10:43,867 EST DEBUG sql.PreparedStatement [WorkflowManager_10027887_55858397233_55868440408_2000] - {pstm-11499549} Types: [java.lang.Long, java.lang.Long]

0 Karma
Reply

493669
Super Champion
‎03-06-2018 07:47 AM

are you using standalone or clustered environment?
and where you have placed props.conf in Heavy forwarder or indexer?

0 Karma
Reply

sarnagar
Contributor
‎03-06-2018 08:00 AM

I'm using clustered env.
I've placed props n transforms on heavy forwarder

0 Karma
Reply

493669
Super Champion
‎03-07-2018 09:57 AM

ok then try this in props.conf:

[sourcetypename]
TRANSFORMS-null= setnull

in transforms.conf

[setnull]
REGEX =(?i)Committing\sall\stransactions\susing\s\(.*\)\stransaction\smanager
DEST_KEY = queue
FORMAT = nullQueue

try this and if it works then add remaining regex...

after making changes restart forwarder

0 Karma
Reply
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...