How do you create a regex that keeps only specific events?

fisuser1 — Thu, 29 Nov 2018 21:35:59 GMT

I'm looking to send junk data to nullque on our heavy forwarder and I only want to key in on specific events in the raw data. I'm looking for a regex to only forward data that contains events below. I'm looking to key on the first few events, since the junk data does not contain the piped ERROR event.

So, if event contains:
"ERROR [WebContainer : 13] [2018-11-29 13:44:23,800] log.UatErrorLogger - |ERROR|"

I want to forward all event data to the indexers. Should we key on "log.UatErrorLogger - |ERROR|"?

Keep:

ERROR [WebContainer : 13] [2018-11-29 13:44:23,800] log.UatErrorLogger - |ERROR|2018-11-29 13:44:23.800 - CST|112|P112736|ERROR|||9bb9e341-bcc4-4902-832d-74c0764237e7||COMPLETED|server14.prod.localserver14||-2|SRM-44116A-MSG|Thread[WebContainer : 13,5,main]|2018-11-29 13:44:23.800 - CST||null|10.237.165.50|||IPV6|SRM|RELEASE|WAS8|BUSINESS_TIER|2.0|UNK|||||||||

Re: How do you create a regex that keeps only specific events?

somesoni2 — Thu, 29 Nov 2018 21:43:07 GMT

Can you add samples of what dhould be discarded..

Re: How do you create a regex that keeps only specific events?

ddrillic — Thu, 29 Nov 2018 21:52:29 GMT

It seems that we need to keep - ^.*(\|ERROR\|)

Re: How do you create a regex that keeps only specific events?

fisuser1 — Fri, 30 Nov 2018 01:20:02 GMT

this is what will need to be sent to nullque

ERROR [WebContainer : 9] [2018-11-29 19:11:54,023] log.UatErrorLogger - 6008a93a-ceae-4ff2-a0cb-79fe6371cb4c
java.lang.NumberFormatException: empty String
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1855)
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:135)
at java.lang.Float.parseFloat(Float.java:462)
at com.metavante.uat.rulescustomization.shared.runtime.DataTypeValue.(DataTypeValue.java:60)
at com.metavante.uat.rulescustomization.shared.runtime.BaseFnRuleImpl.getDataValue(BaseFnRuleImpl.java:475)
at com.metavante.uat.rulescustomization.shared.runtime.generatedrules.AuthoredRule1325000000094104.executeRule(AuthoredRule1325000000094104.java:24)
at com.metavante.uat.rulescustomization.shared.runtime.BaseFnRuleImpl.doExecuteRule(BaseFnRuleImpl.java:73)
at com.metavante.uat.rulescustomization.shared.calcmgr.JavaCalcMgrContextImpl.performCalcs(JavaCalcMgrContextImpl.java:435)
at com.metavante.uat.rulescustomization.shared.calcmgr.JavaCalcMgrContextImpl.runAllCalcs(JavaCalcMgrContextImpl.java:205)
at com.metavante.uat.rulescustomization.shared.calcmgr.JavaCalcMgrContextImpl.runAllCalcs(JavaCalcMgrContextImpl.java:160)
at com.metavante.dx.services.common.utils.UATRulesHelper.executeRules(UATRulesHelper.java:328)
at com.metavante.dx.services.common.handlers.WorkflowRulesHandler.executeHandler(WorkflowRulesHandler.java:178)
at com.metavante.dx.services.flow.processor.SequenceProcessor.executeHandler(SequenceProcessor.java:159)
at com.metavante.dx.services.flow.processor.SequenceProcessor.executeWorkFlow(SequenceProcessor.java:125)
at com.metavante.dx.services.flow.processor.SequenceProcessor.processSequence(SequenceProcessor.java:63)
at com.metavante.eds.los.services.LOSBusinessServiceBase.execute(LOSBusinessServiceBase.java:204)
at com.metavante.eds.los.services.LOSBusinessServicesImpl.losRequestDecision(LOSBusinessServicesImpl.java:315)
at com.metavante.eds.los.services.LOSBusinessServicesImpl$$FastClassByCGLIB$$157720c2.invoke()
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org.springframework.security.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:66)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)
at com.metavante.eds.los.services.LOSBusinessServicesImpl$$EnhancerByCGLIB$$658d4a42.losRequestDecision()
at sun.reflect.GeneratedMethodAccessor745.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
at com.sun.proxy.$Proxy52.losRequestDecision(Unknown Source)
at sun.reflect.GeneratedMethodAccessor745.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.codehaus.xfire.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:59)
at org.codehaus.xfire.service.binding.ServiceInvocationHandler.sendMessage(ServiceInvocationHandler.java:320)
at org.codehaus.xfire.service.binding.ServiceInvocationHandler$1.run(ServiceInvocationHandler.java:86)
at org.codehaus.xfire.service.binding.ServiceInvocationHandler.execute(ServiceInvocationHandler.java:134)
at org.codehaus.xfire.service.binding.ServiceInvocationHandler.invoke(ServiceInvocationHandler.java:109)
at org.codehaus.xfire.handler.HandlerPipeline.invoke(HandlerPipeline.java:131)
at org.codehaus.xfire.transport.DefaultEndpoint.onReceive(DefaultEndpoint.java:64)
at org.codehaus.xfire.transport.AbstractChannel.receive(AbstractChannel.java:38)
at org.codehaus.xfire.transport.http.XFireServletController.invoke(XFireServletController.java:304)
at org.codehaus.xfire.transport.http.XFireServletController.doService(XFireServletController.java:129)
at org.codehaus.xfire.spring.remoting.XFireServletControllerAdapter.handleRequest(XFireServletControllerAdapter.java:67)
at org.codehaus.xfire.spring.remoting.XFireExporter.handleRequest(XFireExporter.java:48)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1235)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:143)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:96)
at com.metavante.dx.filter.ContextCleaningFilter.doFilter(ContextCleaningFilter.java:57)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:197)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:969)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1109)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:82)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:963)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:382)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:532)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:318)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:88)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)

or
ERROR [WebContainer : 9] [2018-11-29 19:09:38,620] log.UatErrorLogger - 4915e949-5603-460b-9b05-3dd0700d3077
java.lang.NullPointerException
at com.metavante.uat.rulescustomization.shared.calcmgr.JavaCalcMgrContextImpl.runSelectedCalcs(JavaCalcMgrContextImpl.java:108)
at com.metavante.uat.rulescustomization.shared.calcmgr.JavaCalcMgrContextImpl.runCalc(JavaCalcMgrContextImpl.java:79)
at com.metavante.dx.services.common.utils.UATRulesHelper.executeRules(UATRulesHelper.java:333)
at com.metavante.dx.services.common.handlers.StepNavOffRulesHandler.excecuteStepNavOffRules(StepNavOffRulesHandler.java:108)
at com.metavante.dx.services.common.handlers.StepNavOffRulesBaseHandler.executeHandler(StepNavOffRulesBaseHandler.java:91)
at com.metavante.dx.services.flow.processor.SequenceProcessor.executeHandler(SequenceProcessor.java:159)
at com.metavante.dx.services.flow.processor.SequenceProcessor.executeWorkFlow(SequenceProcessor.java:125)
at com.metavante.dx.services.flow.processor.SequenceProcessor.processSequence(SequenceProcessor.java:63)
at com.metavante.eds.los.services.LOSBusinessServiceBase.execute(LOSBusinessServiceBase.java:204)
at com.metavante.eds.los.services.LOSBusinessServicesImpl.losEnterApplicants(LOSBusinessServicesImpl.java:263)
at com.metavante.eds.los.services.LOSBusinessServicesImpl$$FastClassByCGLIB$$157720c2.invoke()
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)

Re: How do you create a regex that keeps only specific events?

fisuser1 — Fri, 30 Nov 2018 01:23:19 GMT

this seems to work. I will test some scenarios and update in the AM. Thank you both!

Re: How do you create a regex that keeps only specific events?

mydog8it — Wed, 27 Feb 2019 22:00:22 GMT

In props.conf on Heavy Forwarder add the following:
[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

In the transforms.conf add the following:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = log\.UatErrorLogger\s\-\s\|ERROR\|
DEST_KEY = queue
FORMAT = indexQueue

topic Re: How do you create a regex that keeps only specific events? in Splunk Search

How do you create a regex that keeps only specific events?

Re: How do you create a regex that keeps only specific events?

Re: How do you create a regex that keeps only specific events?

Re: How do you create a regex that keeps only specific events?

Re: How do you create a regex that keeps only specific events?

Re: How do you create a regex that keeps only specific events?