Splunk Search
×

Join the Conversation

Without signing in, you're just watching from the sidelines. Sign in or Register to connect, share, and be part of the Splunk Community.
Ask a Question

How do you create a regex that keeps only specific events?

fisuser1
Contributor
‎11-29-2018 01:35 PM

I'm looking to send junk data to nullque on our heavy forwarder and I only want to key in on specific events in the raw data. I'm looking for a regex to only forward data that contains events below. I'm looking to key on the first few events, since the junk data does not contain the piped ERROR event.

So, if event contains:
"ERROR [WebContainer : 13] [2018-11-29 13:44:23,800] log.UatErrorLogger - |ERROR|"

I want to forward all event data to the indexers. Should we key on "log.UatErrorLogger - |ERROR|"?

Keep:

ERROR [WebContainer : 13] [2018-11-29 13:44:23,800] log.UatErrorLogger - |ERROR|2018-11-29 13:44:23.800 - CST|112|P112736|ERROR|||9bb9e341-bcc4-4902-832d-74c0764237e7||COMPLETED|server14.prod.localserver14||-2|SRM-44116A-MSG|Thread[WebContainer : 13,5,main]|2018-11-29 13:44:23.800 - CST||null|10.237.165.50|||IPV6|SRM|RELEASE|WAS8|BUSINESS_TIER|2.0|UNK|||||||||
Tags (2)
0 Karma
Reply

mydog8it
Builder
‎02-27-2019 02:00 PM

In props.conf on Heavy Forwarder add the following:
[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

In the transforms.conf add the following:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = log\.UatErrorLogger\s\-\s\|ERROR\|
DEST_KEY = queue
FORMAT = indexQueue
0 Karma
Reply

somesoni2
Revered Legend
‎11-29-2018 01:43 PM

Can you add samples of what dhould be discarded..

0 Karma
Reply

fisuser1
Contributor
‎11-29-2018 05:20 PM

this is what will need to be sent to nullque

ERROR [WebContainer : 9] [2018-11-29 19:11:54,023] log.UatErrorLogger - 6008a93a-ceae-4ff2-a0cb-79fe6371cb4c
java.lang.NumberFormatException: empty String
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1855)
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:135)
at java.lang.Float.parseFloat(Float.java:462)
at com.metavante.uat.rulescustomization.shared.runtime.DataTypeValue.(DataTypeValue.java:60)
at com.metavante.uat.rulescustomization.shared.runtime.BaseFnRuleImpl.getDataValue(BaseFnRuleImpl.java:475)
at com.metavante.uat.rulescustomization.shared.runtime.generatedrules.AuthoredRule1325000000094104.executeRule(AuthoredRule1325000000094104.java:24)
at com.metavante.uat.rulescustomization.shared.runtime.BaseFnRuleImpl.doExecuteRule(BaseFnRuleImpl.java:73)
at com.metavante.uat.rulescustomization.shared.calcmgr.JavaCalcMgrContextImpl.performCalcs(JavaCalcMgrContextImpl.java:435)
at com.metavante.uat.rulescustomization.shared.calcmgr.JavaCalcMgrContextImpl.runAllCalcs(JavaCalcMgrContextImpl.java:205)
at com.metavante.uat.rulescustomization.shared.calcmgr.JavaCalcMgrContextImpl.runAllCalcs(JavaCalcMgrContextImpl.java:160)
at com.metavante.dx.services.common.utils.UATRulesHelper.executeRules(UATRulesHelper.java:328)
at com.metavante.dx.services.common.handlers.WorkflowRulesHandler.executeHandler(WorkflowRulesHandler.java:178)
at com.metavante.dx.services.flow.processor.SequenceProcessor.executeHandler(SequenceProcessor.java:159)
at com.metavante.dx.services.flow.processor.SequenceProcessor.executeWorkFlow(SequenceProcessor.java:125)
at com.metavante.dx.services.flow.processor.SequenceProcessor.processSequence(SequenceProcessor.java:63)
at com.metavante.eds.los.services.LOSBusinessServiceBase.execute(LOSBusinessServiceBase.java:204)
at com.metavante.eds.los.services.LOSBusinessServicesImpl.losRequestDecision(LOSBusinessServicesImpl.java:315)
at com.metavante.eds.los.services.LOSBusinessServicesImpl$$FastClassByCGLIB$$157720c2.invoke()
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org.springframework.security.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:66)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)
at com.metavante.eds.los.services.LOSBusinessServicesImpl$$EnhancerByCGLIB$$658d4a42.losRequestDecision()
at sun.reflect.GeneratedMethodAccessor745.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
at com.sun.proxy.$Proxy52.losRequestDecision(Unknown Source)
at sun.reflect.GeneratedMethodAccessor745.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.codehaus.xfire.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:59)
at org.codehaus.xfire.service.binding.ServiceInvocationHandler.sendMessage(ServiceInvocationHandler.java:320)
at org.codehaus.xfire.service.binding.ServiceInvocationHandler$1.run(ServiceInvocationHandler.java:86)
at org.codehaus.xfire.service.binding.ServiceInvocationHandler.execute(ServiceInvocationHandler.java:134)
at org.codehaus.xfire.service.binding.ServiceInvocationHandler.invoke(ServiceInvocationHandler.java:109)
at org.codehaus.xfire.handler.HandlerPipeline.invoke(HandlerPipeline.java:131)
at org.codehaus.xfire.transport.DefaultEndpoint.onReceive(DefaultEndpoint.java:64)
at org.codehaus.xfire.transport.AbstractChannel.receive(AbstractChannel.java:38)
at org.codehaus.xfire.transport.http.XFireServletController.invoke(XFireServletController.java:304)
at org.codehaus.xfire.transport.http.XFireServletController.doService(XFireServletController.java:129)
at org.codehaus.xfire.spring.remoting.XFireServletControllerAdapter.handleRequest(XFireServletControllerAdapter.java:67)
at org.codehaus.xfire.spring.remoting.XFireExporter.handleRequest(XFireExporter.java:48)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1235)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:143)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:96)
at com.metavante.dx.filter.ContextCleaningFilter.doFilter(ContextCleaningFilter.java:57)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:197)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:969)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1109)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:82)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:963)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:382)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:532)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:318)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:88)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)

or
ERROR [WebContainer : 9] [2018-11-29 19:09:38,620] log.UatErrorLogger - 4915e949-5603-460b-9b05-3dd0700d3077
java.lang.NullPointerException
at com.metavante.uat.rulescustomization.shared.calcmgr.JavaCalcMgrContextImpl.runSelectedCalcs(JavaCalcMgrContextImpl.java:108)
at com.metavante.uat.rulescustomization.shared.calcmgr.JavaCalcMgrContextImpl.runCalc(JavaCalcMgrContextImpl.java:79)
at com.metavante.dx.services.common.utils.UATRulesHelper.executeRules(UATRulesHelper.java:333)
at com.metavante.dx.services.common.handlers.StepNavOffRulesHandler.excecuteStepNavOffRules(StepNavOffRulesHandler.java:108)
at com.metavante.dx.services.common.handlers.StepNavOffRulesBaseHandler.executeHandler(StepNavOffRulesBaseHandler.java:91)
at com.metavante.dx.services.flow.processor.SequenceProcessor.executeHandler(SequenceProcessor.java:159)
at com.metavante.dx.services.flow.processor.SequenceProcessor.executeWorkFlow(SequenceProcessor.java:125)
at com.metavante.dx.services.flow.processor.SequenceProcessor.processSequence(SequenceProcessor.java:63)
at com.metavante.eds.los.services.LOSBusinessServiceBase.execute(LOSBusinessServiceBase.java:204)
at com.metavante.eds.los.services.LOSBusinessServicesImpl.losEnterApplicants(LOSBusinessServicesImpl.java:263)
at com.metavante.eds.los.services.LOSBusinessServicesImpl$$FastClassByCGLIB$$157720c2.invoke()
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)

0 Karma
Reply

ddrillic
Ultra Champion
‎11-29-2018 01:52 PM

It seems that we need to keep - ^.*(\|ERROR\|)

Reply

fisuser1
Contributor
‎11-29-2018 05:23 PM

this seems to work. I will test some scenarios and update in the AM. Thank you both!

0 Karma
Reply
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...