Splunk Dev

why I am unable to extract the fields from error log using IFX

nagaraju_chitta
Path Finder

I have an error log as follows. would like to extract the
ERROR and Caused by from the below log.When I try to extract from IFX getting error stated "The extraction failed. If you are extracting multiple fields, try removing one or more fields. Start with extractions that are embedded within longer text strings". Am I doing something wrong?attached screenshot as well.
Sample data:
2017-09-06 11:12:38,415 [] [main] ERROR abc.connectivity.core.cdi.camel.CdiCamelContext: Error starting CamelContext()
org.apache.camel.RuntimeCamelException: org.apache.camel.FailedToCreateRouteException: Failed to create route fixmlToMxmlRouteStart: Route(fixmlToMxmlRouteStart)[[From[jms-tibco:queue:{{jms.inp... because of Failed to resolve endpoint: concurrentConsumers=%5B%25+JMS..JMS_Input_Consumers+%25%5D&transacted=true&transactionManager=%23jtaTransactionManager&useMessageIDAsCorrelationID=true due to: Error during type conversion from type: java.lang.String to the required type: int with value [% JMS..JMS_Input_Consumers %] due java.lang.NumberFormatException: For input string: "[% JMS..JMS_Input_Consumers %]"
at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1642)
... 40 lines omitted ...
at abc.connectivity.client.ConnectivityRunnerBoot.main(ConnectivityRunnerBoot.java:277)
Caused by: org.apache.camel.FailedToCreateRouteException: Failed to create route fixmlToMxmlRouteStart: Route(fixmlToMxmlRouteStart)[[From[jms-tibco:queue:{{jms.inp... because of Failed to resolve endpoint: ?concurrentConsumers=%5B%25+JMS..JMS_Input_Consumers+%25%5D&transacted=true&transactionManager=%23jtaTransactionManager&useMessageIDAsCorrelationID=true due to: Error during type conversion from type: java.lang.String to the required type: int with value [% JMS..JMS_Input_Consumers %] due java.lang.NumberFormatException: For input string: "[% JMS..JMS_Input_Consumers %]"
at org.apache.camel.model.RouteDefinition.addRoutes(RouteDefinition.java:201)
at org.apache.camel.impl.DefaultCamelContext.startRoute(DefaultCamelContext.java:947)
... 9 lines omitted ...
at abc.connectivity.core.cdi.camel.CdiCamelContext.start(CdiCamelContext.java:99)
... 40 more
Caused by: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: aaaa://queue?concurrentConsumers=%5B%25+JMS..JMS_Input_Consumers+%25%5D&transacted=true&transactionManager=%23jtaTransactionManager&useMessageIDAsCorrelationID=true due to: Error during type conversion from type: java.lang.String to the required type: int with value [% JMS..JMS_Input_Consumers %] due java.lang.NumberFormatException: For input string: "[% JMS..JMS_Input_Consumers %]"
at org.apache.camel.impl.DefaultCamelContext.getEndpoint(DefaultCamelContext.java:590)
at org.apache.camel.util.CamelContextHelper.getMandatoryEndpoint(CamelContextHelper.java:79)
at org.apache.camel.model.RouteDefinition.resolveEndpoint(RouteDefinition.java:211)
... 4 lines omitted ...
at org.apache.camel.model.RouteDefinition.addRoutes(RouteDefinition.java:1052)
at org.apache.camel.model.RouteDefinition.addRoutes(RouteDefinition.java:196)
... 51 more
Caused by: org.apache.camel.TypeConversionException: Error during type conversion from type: java.lang.String to the required type: int with value [% JMS..JMS_Input_Consumers %] due java.lang.NumberFormatException: For input string: "[% JMS..JMS_Input_Consumers %]"
at org.apache.camel.impl.converter.BaseTypeConverterRegistry.createTypeConversionException(BaseTypeConverterRegistry.java:610)
... 20 lines omitted ...
Caused by: org.apache.camel.RuntimeCamelException: java.lang.NumberFormatException: For input string: "[% JMS..JMS_Input_Consumers %]"
... 6 lines omitted ...
Caused by: java.lang.NumberFormatException: For input string: "[% JMS..JMS_Input_Consumers %]"

alt text

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi nagaraju_chittathuru,
use a regex like this

(?ms)ERROR\s(?<ERROR>.*?)\s+Caused\sby:\s+(?<Caused_by>.*)

I don't know what are fields you like.

Bye.
Giuseppe

View solution in original post

sbbadri
Motivator

@nagaraju_chittathuru

That error is because, splunk should have already extracted the value and put it some field. So thatt you can't create new field out of it. To resolve this issue, on top right corner on your IFX screen there is drop down button named existing fields.Click on that uncheck the fields and try to extract your required field.

i hope it helps

0 Karma

nagaraju_chitta
Path Finder

@sbbadri...I have already done that ...still the error exist and then I posted the screenshot

0 Karma

nagaraju_chitta
Path Finder

@cusello...thanks for quick turnaround . with provided regex this is extracting the error. but more precisely looking for the below fields.any suggestion would be appreciated. tried with split and mvindex to get the cause by and succeeded and I am not sure weather that is a right approach.
1. timestamp
2. Error (example: [ibus] [main] ERROR testapp.connectivity.core.cdi.camel.CdiCamelContext: Error starting CamelContext(ibus))
3. Description: (example: Failed to create route xmlToMxmlRouteStart: Route(xmlToMxmlRouteStart)[[From[jms-tibco:queue:{{jms.inp... because of Failed to resolve endpoint: jms-tibco://queue:testQueue?concurrentConsumers=%5B%25+JMS.TBUS.JMS_Input_Consumers+%25%5D&transacted=true&transactionManager=%)
4. Caused By count
5. Caused by Description(cause1,cause2,cause3,etc)

Sample data:
2017-09-06 11:12:38,415 [ibus] [main] ERROR testapp.connectivity.core.cdi.camel.CdiCamelContext: Error starting CamelContext(ibus)
org.apache.camel.RuntimeCamelException: org.apache.camel.FailedToCreateRouteException: Failed to create route xmlToMxmlRouteStart: Route(xmlToMxmlRouteStart)[[From[jms-tibco:queue:{{jms.inp... because of Failed to resolve endpoint: jms-tibco://queue:testQueue?concurrentConsumers=%5B%25+JMS.TBUS.JMS_Input_Consumers+%25%5D&transacted=true&transactionManager=%23jtaTransactionManager&useMessageIDAsCorrelationID=true due to: Error during type conversion from type: java.lang.String to the required type: int with value [% JMS.TBUS.JMS_Input_Consumers %] due java.lang.NumberFormatException: For input string: "[% JMS.TBUS.JMS_Input_Consumers %]"
at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1642)
at testapp.connectivity.core.cdi.camel.CdiCamelContext.start(CdiCamelContext.java:101)
at testapp.connectivity.core.cdi.camel.CdiCamelContext.startCamelContext(CdiCamelContext.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88)
at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78)
at org.jboss.weld.injection.MethodInvocationStrategy$DefaultMethodInvocationStrategy.invoke(MethodInvocationStrategy.java:109)
at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:306)
at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:284)
at org.jboss.weld.event.ObserverMethodImpl.notify(ObserverMethodImpl.java:262)
at org.jboss.weld.event.ObserverNotifier.notifySyncObservers(ObserverNotifier.java:271)
at org.jboss.weld.event.ObserverNotifier.notify(ObserverNotifier.java:260)
at org.jboss.weld.event.EventImpl.fire(EventImpl.java:89)
at testapp.connectivity.container.core.ManagedService.doStart(ManagedService.java:315)
at testapp.connectivity.container.common.lifecycle.LifecycleSupport.start(LifecycleSupport.java:51)
at testapp.connectivity.runtime.mx.server.MxServiceServer_3_1_38.doStart(MxServiceServer_3_1_38.java:109)
at testapp.connectivity.container.common.lifecycle.LifecycleSupport.start(LifecycleSupport.java:51)
at testapp.connectivity.runtime.registry.XmlServerRegistry_3_1_38.registerService(XmlServerRegistry_3_1_38.java:110)
at testapp.connectivity.runtime.registry.JmxRegistry.register(JmxRegistry.java:84)
at testapp.connectivity.runtime.registry.FileServerRegistry_3_1_0.register(FileServerRegistry_3_1_0.java:20)
at testapp.connectivity.container.core.ManagedService.(ManagedService.java:224)
at testapp.connectivity.container.core.ManagedService.(ManagedService.java:152)
at testapp.connectivity.container.core.ManagedServiceContainer.startService(ManagedServiceContainer.java:158)
at testapp.connectivity.container.core.ManagedServiceContainer.startManagedInterface(ManagedServiceContainer.java:301)
at testapp.connectivity.container.core.ManagedServiceContainer.startCommand(ManagedServiceContainer.java:295)
at testapp.connectivity.container.core.ManagedServiceContainer.executeCommand(ManagedServiceContainer.java:198)
at testapp.connectivity.runner.Runner.executeCommand(Runner.java:137)
at testapp.connectivity.runner.MxRunner.executeCommand(MxRunner.java:36)
at testapp.connectivity.runner.Runner.bootstrapRunner(Runner.java:249)
at testapp.connectivity.runner.Runner.bootstrapRunner(Runner.java:103)
at testapp.connectivity.runner.MxRunner.bootstrapRunner(MxRunner.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at testapp.connectivity.client.ConnectivityRunnerBoot.startRunner(ConnectivityRunnerBoot.java:801)
at testapp.connectivity.client.ConnectivityRunnerBoot.startInterface(ConnectivityRunnerBoot.java:753)
at testapp.connectivity.client.ConnectivityRunnerBoot.(ConnectivityRunnerBoot.java:141)
at testapp.connectivity.client.ConnectivityRunnerBoot.main(ConnectivityRunnerBoot.java:277)
Caused by: org.apache.camel.FailedToCreateRouteException: Failed to create route xmlToMxmlRouteStart: Route(xmlToMxmlRouteStart)[[From[jms-tibco:queue:{{jms.inp... because of Failed to resolve endpoint: jms-tibco://queue:testQueue?concurrentConsumers=%5B%25+JMS.TBUS.JMS_Input_Consumers+%25%5D&transacted=true&transactionManager=%23jtaTransactionManager&useMessageIDAsCorrelationID=true due to: Error during type conversion from type: java.lang.String to the required type: int with value [% JMS.TBUS.JMS_Input_Consumers %] due java.lang.NumberFormatException: For input string: "[% JMS.TBUS.JMS_Input_Consumers %]"
at org.apache.camel.model.RouteDefinition.addRoutes(RouteDefinition.java:201)
at org.apache.camel.impl.DefaultCamelContext.startRoute(DefaultCamelContext.java:947)
at org.apache.camel.impl.DefaultCamelContext.startRouteDefinitions(DefaultCamelContext.java:3258)
at org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:2981)
at org.apache.camel.impl.DefaultCamelContext.access$000(DefaultCamelContext.java:175)
at org.apache.camel.impl.DefaultCamelContext$2.call(DefaultCamelContext.java:2812)
at org.apache.camel.impl.DefaultCamelContext$2.call(DefaultCamelContext.java:2808)
at org.apache.camel.impl.DefaultCamelContext.doWithDefinedClassLoader(DefaultCamelContext.java:2831)
at org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:2808)
at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61)
at org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:2777)
at testapp.connectivity.core.cdi.camel.CdiCamelContext.start(CdiCamelContext.java:99)
... 40 more
Caused by: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: jms-tibco://queue:testQueue?concurrentConsumers=%5B%25+JMS.TBUS.JMS_Input_Consumers+%25%5D&transacted=true&transactionManager=%23jtaTransactionManager&useMessageIDAsCorrelationID=true due to: Error during type conversion from type: java.lang.String to the required type: int with value [% JMS.TBUS.JMS_Input_Consumers %] due java.lang.NumberFormatException: For input string: "[% JMS.TBUS.JMS_Input_Consumers %]"
at org.apache.camel.impl.DefaultCamelContext.getEndpoint(DefaultCamelContext.java:590)
at org.apache.camel.util.CamelContextHelper.getMandatoryEndpoint(CamelContextHelper.java:79)
at org.apache.camel.model.RouteDefinition.resolveEndpoint(RouteDefinition.java:211)
at org.apache.camel.impl.DefaultRouteContext.resolveEndpoint(DefaultRouteContext.java:107)
at org.apache.camel.impl.DefaultRouteContext.resolveEndpoint(DefaultRouteContext.java:113)
at org.apache.camel.model.FromDefinition.resolveEndpoint(FromDefinition.java:69)
at org.apache.camel.impl.DefaultRouteContext.getEndpoint(DefaultRouteContext.java:89)
at org.apache.camel.model.RouteDefinition.addRoutes(RouteDefinition.java:1052)
at org.apache.camel.model.RouteDefinition.addRoutes(RouteDefinition.java:196)
... 51 more
Caused by: org.apache.camel.TypeConversionException: Error during type conversion from type: java.lang.String to the required type: int with value [% JMS.TBUS.JMS_Input_Consumers %] due java.lang.NumberFormatException: For input string: "[% JMS.TBUS.JMS_Input_Consumers %]"
at org.apache.camel.impl.converter.BaseTypeConverterRegistry.createTypeConversionException(BaseTypeConverterRegistry.java:610)
at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:177)
at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:156)
at org.apache.camel.util.IntrospectionSupport.convert(IntrospectionSupport.java:622)
at org.apache.camel.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:537)
at org.apache.camel.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:602)
at org.apache.camel.util.IntrospectionSupport.setProperties(IntrospectionSupport.java:459)
at org.apache.camel.util.IntrospectionSupport.setProperties(IntrospectionSupport.java:469)
at org.apache.camel.util.EndpointHelper.setProperties(EndpointHelper.java:256)
at org.apache.camel.impl.DefaultComponent.setProperties(DefaultComponent.java:257)
at org.apache.camel.component.jms.JmsComponent.createEndpoint(JmsComponent.java:886)
at org.apache.camel.impl.DefaultComponent.createEndpoint(DefaultComponent.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.weld.bean.proxy.AbstractBeanInstance.invoke(AbstractBeanInstance.java:38)
at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
at org.jboss.weld.proxies.CamelContextAware$Component$1063300809$Proxy$_$$_WeldClientProxy.createEndpoint(Unknown Source)
at org.apache.camel.impl.DefaultCamelContext.getEndpoint(DefaultCamelContext.java:570)
... 59 more
Caused by: org.apache.camel.RuntimeCamelException: java.lang.NumberFormatException: For input string: "[% JMS.TBUS.JMS_Input_Consumers %]"
at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1642)
at org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1247)
at org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:59)
at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:315)
at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:168)
... 77 more
Caused by: java.lang.NumberFormatException: For input string: "[% JMS.TBUS.JMS_Input_Consumers %]"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.valueOf(Integer.java:766)
at org.apache.camel.converter.ObjectConverter.toInteger(ObjectConverter.java:262)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1243)
... 80 more

0 Karma

lfedak_splunk
Splunk Employee
Splunk Employee

Hi @nagaraju_chittathuru, can you please limit your post to 10,000 characters then convert it to a comment? Please reserve the Answers feature strictly for answers to the original question and use the comments feature for any conversation/comments.

0 Karma

nagaraju_chitta
Path Finder

@Ifedak, apologies...initially tried to post this in the comments and somehow could not be able to submit hence added like this...going forward will follow the best practices to post the question and related comments.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi nagaraju_chittathuru,
use a regex like this

(?ms)ERROR\s(?<ERROR>.*?)\s+Caused\sby:\s+(?<Caused_by>.*)

I don't know what are fields you like.

Bye.
Giuseppe

nagaraju_chitta
Path Finder

Hi cusello...with the below regex I can be able to get the whole error. would like to extract date as well from the provided sample. Also is it a good practice to split the event and can show the portion of the error as a description or would you suggest the description also has to extract using REGEX?

(?ms)ERROR\s(?<ERROR>.*?)\s+Caused\sby:\s+(?<Caused_by>.*)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi nagaraju_chittathuru,
You should define if it's better for you to have the full error message or a part of it: I have an App where customer asked to me to display the full message, instead in another App the same customer asked to me to display only a part of it.

Watching your log, I see that it's easier to extract the full log because it's difficoult to find a rule to identify the part of log to take: e.g. it's easy to extract the first string without spaces after "Caused by" (your first bold string) but it's difficoult to identify the second bold string (there aren't clear identifiers).

Eventually in your dashboard you could put a text box to perform a full text search in error messages that could help your users to use this log.

bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...