Splunk Search
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question

How do you create a regex that keeps only specific events?

fisuser1
Contributor
‎11-29-2018 01:35 PM

I'm looking to send junk data to nullque on our heavy forwarder and I only want to key in on specific events in the raw data. I'm looking for a regex to only forward data that contains events below. I'm looking to key on the first few events, since the junk data does not contain the piped ERROR event.

So, if event contains:
"ERROR [WebContainer : 13] [2018-11-29 13:44:23,800] log.UatErrorLogger - |ERROR|"

I want to forward all event data to the indexers. Should we key on "log.UatErrorLogger - |ERROR|"?

Keep:

ERROR [WebContainer : 13] [2018-11-29 13:44:23,800] log.UatErrorLogger - |ERROR|2018-11-29 13:44:23.800 - CST|112|P112736|ERROR|||9bb9e341-bcc4-4902-832d-74c0764237e7||COMPLETED|server14.prod.localserver14||-2|SRM-44116A-MSG|Thread[WebContainer : 13,5,main]|2018-11-29 13:44:23.800 - CST||null|10.237.165.50|||IPV6|SRM|RELEASE|WAS8|BUSINESS_TIER|2.0|UNK|||||||||
Tags (2)
0 Karma
Reply

mydog8it
Builder
‎02-27-2019 02:00 PM

In props.conf on Heavy Forwarder add the following:
[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

In the transforms.conf add the following:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = log\.UatErrorLogger\s\-\s\|ERROR\|
DEST_KEY = queue
FORMAT = indexQueue
0 Karma
Reply

somesoni2
Revered Legend
‎11-29-2018 01:43 PM

Can you add samples of what dhould be discarded..

0 Karma
Reply

fisuser1
Contributor
‎11-29-2018 05:20 PM

this is what will need to be sent to nullque

ERROR [WebContainer : 9] [2018-11-29 19:11:54,023] log.UatErrorLogger - 6008a93a-ceae-4ff2-a0cb-79fe6371cb4c
java.lang.NumberFormatException: empty String
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1855)
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:135)
at java.lang.Float.parseFloat(Float.java:462)
at com.metavante.uat.rulescustomization.shared.runtime.DataTypeValue.(DataTypeValue.java:60)
at com.metavante.uat.rulescustomization.shared.runtime.BaseFnRuleImpl.getDataValue(BaseFnRuleImpl.java:475)
at com.metavante.uat.rulescustomization.shared.runtime.generatedrules.AuthoredRule1325000000094104.executeRule(AuthoredRule1325000000094104.java:24)
at com.metavante.uat.rulescustomization.shared.runtime.BaseFnRuleImpl.doExecuteRule(BaseFnRuleImpl.java:73)
at com.metavante.uat.rulescustomization.shared.calcmgr.JavaCalcMgrContextImpl.performCalcs(JavaCalcMgrContextImpl.java:435)
at com.metavante.uat.rulescustomization.shared.calcmgr.JavaCalcMgrContextImpl.runAllCalcs(JavaCalcMgrContextImpl.java:205)
at com.metavante.uat.rulescustomization.shared.calcmgr.JavaCalcMgrContextImpl.runAllCalcs(JavaCalcMgrContextImpl.java:160)
at com.metavante.dx.services.common.utils.UATRulesHelper.executeRules(UATRulesHelper.java:328)
at com.metavante.dx.services.common.handlers.WorkflowRulesHandler.executeHandler(WorkflowRulesHandler.java:178)
at com.metavante.dx.services.flow.processor.SequenceProcessor.executeHandler(SequenceProcessor.java:159)
at com.metavante.dx.services.flow.processor.SequenceProcessor.executeWorkFlow(SequenceProcessor.java:125)
at com.metavante.dx.services.flow.processor.SequenceProcessor.processSequence(SequenceProcessor.java:63)
at com.metavante.eds.los.services.LOSBusinessServiceBase.execute(LOSBusinessServiceBase.java:204)
at com.metavante.eds.los.services.LOSBusinessServicesImpl.losRequestDecision(LOSBusinessServicesImpl.java:315)
at com.metavante.eds.los.services.LOSBusinessServicesImpl$$FastClassByCGLIB$$157720c2.invoke()
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org.springframework.security.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:66)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)
at com.metavante.eds.los.services.LOSBusinessServicesImpl$$EnhancerByCGLIB$$658d4a42.losRequestDecision()
at sun.reflect.GeneratedMethodAccessor745.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
at com.sun.proxy.$Proxy52.losRequestDecision(Unknown Source)
at sun.reflect.GeneratedMethodAccessor745.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.codehaus.xfire.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:59)
at org.codehaus.xfire.service.binding.ServiceInvocationHandler.sendMessage(ServiceInvocationHandler.java:320)
at org.codehaus.xfire.service.binding.ServiceInvocationHandler$1.run(ServiceInvocationHandler.java:86)
at org.codehaus.xfire.service.binding.ServiceInvocationHandler.execute(ServiceInvocationHandler.java:134)
at org.codehaus.xfire.service.binding.ServiceInvocationHandler.invoke(ServiceInvocationHandler.java:109)
at org.codehaus.xfire.handler.HandlerPipeline.invoke(HandlerPipeline.java:131)
at org.codehaus.xfire.transport.DefaultEndpoint.onReceive(DefaultEndpoint.java:64)
at org.codehaus.xfire.transport.AbstractChannel.receive(AbstractChannel.java:38)
at org.codehaus.xfire.transport.http.XFireServletController.invoke(XFireServletController.java:304)
at org.codehaus.xfire.transport.http.XFireServletController.doService(XFireServletController.java:129)
at org.codehaus.xfire.spring.remoting.XFireServletControllerAdapter.handleRequest(XFireServletControllerAdapter.java:67)
at org.codehaus.xfire.spring.remoting.XFireExporter.handleRequest(XFireExporter.java:48)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1235)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:143)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:96)
at com.metavante.dx.filter.ContextCleaningFilter.doFilter(ContextCleaningFilter.java:57)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:197)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:969)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1109)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:82)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:963)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:382)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:532)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:318)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:88)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)

or
ERROR [WebContainer : 9] [2018-11-29 19:09:38,620] log.UatErrorLogger - 4915e949-5603-460b-9b05-3dd0700d3077
java.lang.NullPointerException
at com.metavante.uat.rulescustomization.shared.calcmgr.JavaCalcMgrContextImpl.runSelectedCalcs(JavaCalcMgrContextImpl.java:108)
at com.metavante.uat.rulescustomization.shared.calcmgr.JavaCalcMgrContextImpl.runCalc(JavaCalcMgrContextImpl.java:79)
at com.metavante.dx.services.common.utils.UATRulesHelper.executeRules(UATRulesHelper.java:333)
at com.metavante.dx.services.common.handlers.StepNavOffRulesHandler.excecuteStepNavOffRules(StepNavOffRulesHandler.java:108)
at com.metavante.dx.services.common.handlers.StepNavOffRulesBaseHandler.executeHandler(StepNavOffRulesBaseHandler.java:91)
at com.metavante.dx.services.flow.processor.SequenceProcessor.executeHandler(SequenceProcessor.java:159)
at com.metavante.dx.services.flow.processor.SequenceProcessor.executeWorkFlow(SequenceProcessor.java:125)
at com.metavante.dx.services.flow.processor.SequenceProcessor.processSequence(SequenceProcessor.java:63)
at com.metavante.eds.los.services.LOSBusinessServiceBase.execute(LOSBusinessServiceBase.java:204)
at com.metavante.eds.los.services.LOSBusinessServicesImpl.losEnterApplicants(LOSBusinessServicesImpl.java:263)
at com.metavante.eds.los.services.LOSBusinessServicesImpl$$FastClassByCGLIB$$157720c2.invoke()
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)

0 Karma
Reply

ddrillic
Ultra Champion
‎11-29-2018 01:52 PM

It seems that we need to keep - ^.*(\|ERROR\|)

Reply

fisuser1
Contributor
‎11-29-2018 05:23 PM

this seems to work. I will test some scenarios and update in the AM. Thank you both!

0 Karma
Reply
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...