Splunk Search

HTTP Event Collector multi-line merge for a container

markthill
Engager

Using splunk 8.0.2.1

I have a container (spring boot that uses tomcat underneath) that I'm running that I'm attempting to push the contents to the HEC.   I'm starting the container like this:

docker run --name test-spring-boot-app --publish 8080:8080 --log-driver=splunk --log-opt splunk-token=SOME-TOKEN --log-opt splunk-url=http://ec2-someip.compute-1.amazonaws.com:8088 --log-opt splunk-format=inline --log-opt splunk-sourcetype=log4j-test test-spring-boot-app

I can't for the life of me get ingested logs to merge multi-line events.  The Exception in the log below shows up as a single event for every line even though I've tried every combination I can think of to try to get it to merge.  It almost appears that it is ignoring my source type.  I have the token in HEC selected with the log4j-test source type as well.

My log output looks like this:

 

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.3.1.RELEASE)

2020-06-29 19:57:52,828 [main] INFO  com.sss.app.ws.TestSpringBootAppApplication - Starting TestSpringBootAppApplication v0.0.1-SNAPSHOT on 84837ec423e5 with PID 1 (/spring-boot-test.jar started by root in /)
2020-06-29 19:57:52,843 [main] INFO  com.sss.app.ws.TestSpringBootAppApplication - No active profile set, falling back to default profiles: default
2020-06-29 19:57:54,370 [main] INFO  org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8080 (http)
2020-06-29 19:57:54,406 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8080"]
2020-06-29 19:57:54,407 [main] INFO  org.apache.catalina.core.StandardService - Starting service [Tomcat]
2020-06-29 19:57:54,408 [main] INFO  org.apache.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.36]
2020-06-29 19:57:54,520 [main] INFO  org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2020-06-29 19:57:54,520 [main] INFO  org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 1597 ms
2020-06-29 19:57:54,856 [main] INFO  org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Initializing ExecutorService 'applicationTaskExecutor'
2020-06-29 19:57:55,080 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8080"]
2020-06-29 19:57:55,128 [main] INFO  org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8080 (http) with context path ''
2020-06-29 19:57:55,143 [main] INFO  com.sss.app.ws.TestSpringBootAppApplication - Started TestSpringBootAppApplication in 2.877 seconds (JVM running for 4.391)
2020-06-29 19:58:01,670 [http-nio-8080-exec-1] INFO  org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-06-29 19:58:01,670 [http-nio-8080-exec-1] INFO  org.springframework.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcherServlet'
2020-06-29 19:58:01,680 [http-nio-8080-exec-1] INFO  org.springframework.web.servlet.DispatcherServlet - Completed initialization in 10 ms
2020-06-29 19:58:01,807 [http-nio-8080-exec-1] INFO  com.sss.app.ws.controller.TestController - foo bar log: true
2020-06-29 19:58:01,807 [http-nio-8080-exec-1] INFO  com.sss.app.ws.controller.TestController - The querystring parameter name was supplied as: mark
2020-06-29 19:58:01,807 [http-nio-8080-exec-1] INFO  com.sss.app.ws.controller.TestController - The querystring parameter exc was supplied as: true
2020-06-29 19:58:01,813 [http-nio-8080-exec-1] ERROR org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.Exception: Give me an exception please] with root cause
java.lang.Exception: Give me an exception please
	at com.sss.app.ws.controller.TestController.getTest(TestController.java:47) ~[classes!/:0.0.1-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111-internal]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_111-internal]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111-internal]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111-internal]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]

 

In my props.conf I have log4j-test which looks like:

 

./splunk btool --debug props list log4j-test | more
/home/ubuntu/apps/splunk/etc/system/default/props.conf [log4j-test]
/home/ubuntu/apps/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/home/ubuntu/apps/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/home/ubuntu/apps/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/home/ubuntu/apps/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE = \d\d?:\d\d:\d\d
/home/ubuntu/apps/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/home/ubuntu/apps/splunk/etc/system/default/props.conf CHARSET = UTF-8
/home/ubuntu/apps/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/home/ubuntu/apps/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/home/ubuntu/apps/splunk/etc/system/default/props.conf HEADER_MODE = 
/home/ubuntu/apps/splunk/etc/system/default/props.conf LEARN_MODEL = true
/home/ubuntu/apps/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/home/ubuntu/apps/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/home/ubuntu/apps/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/home/ubuntu/apps/splunk/etc/system/default/props.conf MUST_BREAK_AFTER = 
/home/ubuntu/apps/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER = 
/home/ubuntu/apps/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE = 
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/home/ubuntu/apps/splunk/etc/system/default/props.conf SHOULD_LINEMERGE = true
/home/ubuntu/apps/splunk/etc/system/default/props.conf TRANSFORMS = 
/home/ubuntu/apps/splunk/etc/system/default/props.conf TRUNCATE = 10000
/home/ubuntu/apps/splunk/etc/system/default/props.conf category = Application
/home/ubuntu/apps/splunk/etc/system/default/props.conf description = Test Output produced by any Java 2 Enterprise Edition (J2EE) application server using log4j
/home/ubuntu/apps/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/home/ubuntu/apps/splunk/etc/system/default/props.conf maxDist = 75
/home/ubuntu/apps/splunk/etc/system/default/props.conf priority = 
/home/ubuntu/apps/splunk/etc/system/default/props.conf pulldown_type = true
/home/ubuntu/apps/splunk/etc/system/default/props.conf sourcetype = 

 

Any thoughts would be greatly appreciated.

Labels (1)

markthill
Engager

Docker seems to output each line to splunk (using the --log-driver=splunk) to a separate event because it treats each line as an event.  I have somewhat solved this by using log4j2 to output each log item to json, so it now spits out the entire stacktrace in a json wrapped line.  I'm using:

<JSONLayout compact="true" stacktraceAsString="true" eventEol="true"/>

This works for the most part until I get to searching in splunk.  My logger now outputs something like this for an exception:

{"instant":{"epochSecond":1593534200,"nanoOfSecond":461000000},"thread":"http-nio-8080-exec-1","level":"ERROR","loggerName":"org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet]","message":"Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.Exception: Give me an exception please] with root cause","thrown":{"commonElementCount":0,"localizedMessage":"Give me an exception please","message":"Give me an exception please","name":"java.lang.Exception","extendedStackTrace":"java.lang.Exception: Give me an exception please\n\tat com.sss.app.ws.controller.TestController.getTest(TestController.java:47) ~[classes!/:0.0.1-SNAPSHOT]\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111-internal]\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_111-internal]\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111-internal]\n\tat java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111-internal]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111-internal]\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111-internal]\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat java.lang.Thread.run(Thread.java:745) [?:1.8.0_111-internal]\n"},"endOfBatch":false,"loggerFqcn":"java.util.logging.Logger","threadId":117,"threadPriority":5}

 But now when I search for it in splunk it escapes all the \n\t so it doesn't display properly in the search results.  Working on the next problem....

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...