Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About markthill
markthill
Observer
Member since:
Monday
yesterday
Community Statistics
| Posts | 2 |
| Solutions | 0 |
| Karma Given | 0 |
| Karma Received | 0 |
| Member Since | 06-29-2020 |
Activity Feed
- Posted Re: HTTP Event Collector multi-line merge for a container on Splunk Search. yesterday
- Posted HTTP Event Collector multi-line merge for a container on Splunk Search. Monday
yesterday
Docker seems to output each line to splunk (using the --log-driver=splunk) to a separate event because it treats each line as an event. I have somewhat solved this by using log4j2 to output each log item to json, so it now spits out the entire stacktrace in a json wrapped line. I'm using: < JSONLayout compact = "true" stacktraceAsString = "true" eventEol = "true" /> This works for the most part until I get to searching in splunk. My logger now outputs something like this for an exception: {"instant":{"epochSecond":1593534200,"nanoOfSecond":461000000},"thread":"http-nio-8080-exec-1","level":"ERROR","loggerName":"org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet]","message":"Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.Exception: Give me an exception please] with root cause","thrown":{"commonElementCount":0,"localizedMessage":"Give me an exception please","message":"Give me an exception please","name":"java.lang.Exception","extendedStackTrace":"java.lang.Exception: Give me an exception please\n\tat com.sss.app.ws.controller.TestController.getTest(TestController.java:47) ~[classes!/:0.0.1-SNAPSHOT]\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111-internal]\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_111-internal]\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111-internal]\n\tat java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111-internal]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111-internal]\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111-internal]\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.36.jar!/:9.0.36]\n\tat java.lang.Thread.run(Thread.java:745) [?:1.8.0_111-internal]\n"},"endOfBatch":false,"loggerFqcn":"java.util.logging.Logger","threadId":117,"threadPriority":5} But now when I search for it in splunk it escapes all the \n\t so it doesn't display properly in the search results. Working on the next problem....
... View more
Monday
Using splunk 8.0.2.1 I have a container (spring boot that uses tomcat underneath) that I'm running that I'm attempting to push the contents to the HEC. I'm starting the container like this: docker run --name test-spring-boot-app --publish 8080:8080 --log-driver=splunk --log-opt splunk-token=SOME-TOKEN --log-opt splunk-url=http://ec2-someip.compute-1.amazonaws.com:8088 --log-opt splunk-format=inline --log-opt splunk-sourcetype=log4j-test test-spring-boot-app I can't for the life of me get ingested logs to merge multi-line events. The Exception in the log below shows up as a single event for every line even though I've tried every combination I can think of to try to get it to merge. It almost appears that it is ignoring my source type. I have the token in HEC selected with the log4j-test source type as well. My log output looks like this: . ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.3.1.RELEASE)
2020-06-29 19:57:52,828 [main] INFO com.sss.app.ws.TestSpringBootAppApplication - Starting TestSpringBootAppApplication v0.0.1-SNAPSHOT on 84837ec423e5 with PID 1 (/spring-boot-test.jar started by root in /)
2020-06-29 19:57:52,843 [main] INFO com.sss.app.ws.TestSpringBootAppApplication - No active profile set, falling back to default profiles: default
2020-06-29 19:57:54,370 [main] INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8080 (http)
2020-06-29 19:57:54,406 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8080"]
2020-06-29 19:57:54,407 [main] INFO org.apache.catalina.core.StandardService - Starting service [Tomcat]
2020-06-29 19:57:54,408 [main] INFO org.apache.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.36]
2020-06-29 19:57:54,520 [main] INFO org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2020-06-29 19:57:54,520 [main] INFO org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 1597 ms
2020-06-29 19:57:54,856 [main] INFO org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Initializing ExecutorService 'applicationTaskExecutor'
2020-06-29 19:57:55,080 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8080"]
2020-06-29 19:57:55,128 [main] INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8080 (http) with context path ''
2020-06-29 19:57:55,143 [main] INFO com.sss.app.ws.TestSpringBootAppApplication - Started TestSpringBootAppApplication in 2.877 seconds (JVM running for 4.391)
2020-06-29 19:58:01,670 [http-nio-8080-exec-1] INFO org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-06-29 19:58:01,670 [http-nio-8080-exec-1] INFO org.springframework.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcherServlet'
2020-06-29 19:58:01,680 [http-nio-8080-exec-1] INFO org.springframework.web.servlet.DispatcherServlet - Completed initialization in 10 ms
2020-06-29 19:58:01,807 [http-nio-8080-exec-1] INFO com.sss.app.ws.controller.TestController - foo bar log: true
2020-06-29 19:58:01,807 [http-nio-8080-exec-1] INFO com.sss.app.ws.controller.TestController - The querystring parameter name was supplied as: mark
2020-06-29 19:58:01,807 [http-nio-8080-exec-1] INFO com.sss.app.ws.controller.TestController - The querystring parameter exc was supplied as: true
2020-06-29 19:58:01,813 [http-nio-8080-exec-1] ERROR org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.Exception: Give me an exception please] with root cause
java.lang.Exception: Give me an exception please
at com.sss.app.ws.controller.TestController.getTest(TestController.java:47) ~[classes!/:0.0.1-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111-internal]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_111-internal]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111-internal]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111-internal]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE] In my props.conf I have log4j-test which looks like: ./splunk btool --debug props list log4j-test | more
/home/ubuntu/apps/splunk/etc/system/default/props.conf [log4j-test]
/home/ubuntu/apps/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/home/ubuntu/apps/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/home/ubuntu/apps/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/home/ubuntu/apps/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE = \d\d?:\d\d:\d\d
/home/ubuntu/apps/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/home/ubuntu/apps/splunk/etc/system/default/props.conf CHARSET = UTF-8
/home/ubuntu/apps/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/home/ubuntu/apps/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/home/ubuntu/apps/splunk/etc/system/default/props.conf HEADER_MODE =
/home/ubuntu/apps/splunk/etc/system/default/props.conf LEARN_MODEL = true
/home/ubuntu/apps/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/home/ubuntu/apps/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/home/ubuntu/apps/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/home/ubuntu/apps/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/home/ubuntu/apps/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/home/ubuntu/apps/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/home/ubuntu/apps/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/home/ubuntu/apps/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/home/ubuntu/apps/splunk/etc/system/default/props.conf SHOULD_LINEMERGE = true
/home/ubuntu/apps/splunk/etc/system/default/props.conf TRANSFORMS =
/home/ubuntu/apps/splunk/etc/system/default/props.conf TRUNCATE = 10000
/home/ubuntu/apps/splunk/etc/system/default/props.conf category = Application
/home/ubuntu/apps/splunk/etc/system/default/props.conf description = Test Output produced by any Java 2 Enterprise Edition (J2EE) application server using log4j
/home/ubuntu/apps/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/home/ubuntu/apps/splunk/etc/system/default/props.conf maxDist = 75
/home/ubuntu/apps/splunk/etc/system/default/props.conf priority =
/home/ubuntu/apps/splunk/etc/system/default/props.conf pulldown_type = true
/home/ubuntu/apps/splunk/etc/system/default/props.conf sourcetype = Any thoughts would be greatly appreciated.
... View more
Labels
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
yesterday
|
