Using splunk 8.0.2.1 I have a container (spring boot that uses tomcat underneath) that I'm running that I'm attempting to push the contents to the HEC. I'm starting the container like this: docker run --name test-spring-boot-app --publish 8080:8080 --log-driver=splunk --log-opt splunk-token=SOME-TOKEN --log-opt splunk-url=http://ec2-someip.compute-1.amazonaws.com:8088 --log-opt splunk-format=inline --log-opt splunk-sourcetype=log4j-test test-spring-boot-app I can't for the life of me get ingested logs to merge multi-line events. The Exception in the log below shows up as a single event for every line even though I've tried every combination I can think of to try to get it to merge. It almost appears that it is ignoring my source type. I have the token in HEC selected with the log4j-test source type as well. My log output looks like this: . ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.3.1.RELEASE)
2020-06-29 19:57:52,828 [main] INFO com.sss.app.ws.TestSpringBootAppApplication - Starting TestSpringBootAppApplication v0.0.1-SNAPSHOT on 84837ec423e5 with PID 1 (/spring-boot-test.jar started by root in /)
2020-06-29 19:57:52,843 [main] INFO com.sss.app.ws.TestSpringBootAppApplication - No active profile set, falling back to default profiles: default
2020-06-29 19:57:54,370 [main] INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8080 (http)
2020-06-29 19:57:54,406 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8080"]
2020-06-29 19:57:54,407 [main] INFO org.apache.catalina.core.StandardService - Starting service [Tomcat]
2020-06-29 19:57:54,408 [main] INFO org.apache.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.36]
2020-06-29 19:57:54,520 [main] INFO org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2020-06-29 19:57:54,520 [main] INFO org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 1597 ms
2020-06-29 19:57:54,856 [main] INFO org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Initializing ExecutorService 'applicationTaskExecutor'
2020-06-29 19:57:55,080 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8080"]
2020-06-29 19:57:55,128 [main] INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8080 (http) with context path ''
2020-06-29 19:57:55,143 [main] INFO com.sss.app.ws.TestSpringBootAppApplication - Started TestSpringBootAppApplication in 2.877 seconds (JVM running for 4.391)
2020-06-29 19:58:01,670 [http-nio-8080-exec-1] INFO org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-06-29 19:58:01,670 [http-nio-8080-exec-1] INFO org.springframework.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcherServlet'
2020-06-29 19:58:01,680 [http-nio-8080-exec-1] INFO org.springframework.web.servlet.DispatcherServlet - Completed initialization in 10 ms
2020-06-29 19:58:01,807 [http-nio-8080-exec-1] INFO com.sss.app.ws.controller.TestController - foo bar log: true
2020-06-29 19:58:01,807 [http-nio-8080-exec-1] INFO com.sss.app.ws.controller.TestController - The querystring parameter name was supplied as: mark
2020-06-29 19:58:01,807 [http-nio-8080-exec-1] INFO com.sss.app.ws.controller.TestController - The querystring parameter exc was supplied as: true
2020-06-29 19:58:01,813 [http-nio-8080-exec-1] ERROR org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.Exception: Give me an exception please] with root cause
java.lang.Exception: Give me an exception please
at com.sss.app.ws.controller.TestController.getTest(TestController.java:47) ~[classes!/:0.0.1-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111-internal]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_111-internal]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111-internal]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111-internal]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE] In my props.conf I have log4j-test which looks like: ./splunk btool --debug props list log4j-test | more
/home/ubuntu/apps/splunk/etc/system/default/props.conf [log4j-test]
/home/ubuntu/apps/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/home/ubuntu/apps/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/home/ubuntu/apps/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/home/ubuntu/apps/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE = \d\d?:\d\d:\d\d
/home/ubuntu/apps/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/home/ubuntu/apps/splunk/etc/system/default/props.conf CHARSET = UTF-8
/home/ubuntu/apps/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/home/ubuntu/apps/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/home/ubuntu/apps/splunk/etc/system/default/props.conf HEADER_MODE =
/home/ubuntu/apps/splunk/etc/system/default/props.conf LEARN_MODEL = true
/home/ubuntu/apps/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/home/ubuntu/apps/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/home/ubuntu/apps/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/home/ubuntu/apps/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/home/ubuntu/apps/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/home/ubuntu/apps/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/home/ubuntu/apps/splunk/etc/system/default/props.conf SHOULD_LINEMERGE = true
/home/ubuntu/apps/splunk/etc/system/default/props.conf TRANSFORMS =
/home/ubuntu/apps/splunk/etc/system/default/props.conf TRUNCATE = 10000
/home/ubuntu/apps/splunk/etc/system/default/props.conf category = Application
/home/ubuntu/apps/splunk/etc/system/default/props.conf description = Test Output produced by any Java 2 Enterprise Edition (J2EE) application server using log4j
/home/ubuntu/apps/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/home/ubuntu/apps/splunk/etc/system/default/props.conf maxDist = 75
/home/ubuntu/apps/splunk/etc/system/default/props.conf priority =
/home/ubuntu/apps/splunk/etc/system/default/props.conf pulldown_type = true
/home/ubuntu/apps/splunk/etc/system/default/props.conf sourcetype = Any thoughts would be greatly appreciated.
... View more