Getting Data In

How to use Splunk to detect custom log pattern when using sleuth for traces?

New Member

Sample log file output

2018-01-29 17:46:35.341  INFO [hello-service,ca62f5d265c65e37,ca62f5d265c65e37,true] 9404 ---  [  XNIO-2 task-1] io.undertow.servlet                       Initializing Spring FrameworkServlet 'dispatcherServlet'
2018-01-29 17:46:35.342  INFO [hello-service,ca62f5d265c65e37,ca62f5d265c65e37,true] 9404 ---  [  XNIO-2 task-1] o.s.web.servlet.DispatcherServlet         FrameworkServlet 'dispatcherServlet': initialization started
2018-01-29 17:46:35.456  INFO [hello-service,ca62f5d265c65e37,ca62f5d265c65e37,true] 9404 ---  [  XNIO-2 task-1] o.s.web.servlet.DispatcherServlet         FrameworkServlet 'dispatcherServlet': initialization completed in 114 ms
2018-01-29 17:46:35.523  INFO [hello-service,ca62f5d265c65e37,ca62f5d265c65e37,true] 9404 ---  [  XNIO-2 task-1]          Received request
2018-01-29 17:46:42.009  INFO [hello-service,f3f5b0389dcdd2e9,f3f5b0389dcdd2e9,false] 9404 ---  [  XNIO-2 task-2]          Received request
2018-01-29 17:46:46.534  INFO [hello-service,7f2fed9d81715a4e,7f2fed9d81715a4e,true] 9404 ---  [  XNIO-2 task-3]          Received request

LOG_PATTERN used in logback-spring.xml configuration file

    "%d{yyyy-MM-dd HH:mm:ss.SSS}
    ${LOG_LEVEL_PATTERN:-%5p} ${PID:- }
    [%15.15t] %-40.40logger{39}  %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"

I am using Spring Cloud Sleuth to generate the traceId in the logs using the above pattern in logback configuration.

When I Import the log file to Splunk, I want Splunk to recognize the fields
timestamp,loglevel,[serviceId,traceId,segmentId,exportToZipkin),processId and so on corresponding to the pattern in the LOG_PATTERN

I want Splunk to recognize because I intend to do aggregate searches in Splunk based on the serviceId,traceId,segmentIdfields logged in the file.

Does Splunk automatically recognize this log pattern or is there a way to teach Splunk to recognize this format so as to enable it to index the new data accordingly, and to provide options to search by the respective fields.

0 Karma


Hi volijaadu,

After indexing data you can use auto extractor of splunk or write an regex to extract required fields.
Use this link to use splunk field extractor:

0 Karma

New Member

I do not want to do it at request/search time as there will be a performance impact. I would like to do it at index time so that searches are faster at search time.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...