props.conf help

[a212830]

Hi,

I have a multi-line feed that I'm having problems with - hoping someone can help me. Here's my props.conf:

MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = 1
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TIME_PREFIX = ^
TRUNCATE = 999999
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\d{4}\s\w{2}\s\d{2}

And, here's some sample data. The break is not occurring on the lines properly. Pretty sure my LINE_BREAKER is wrong. On a side note, if I just let splunk do the work, everything is wonderful, but this is not how we want to eng our systems. Is there any way to tell what splunk is applying for timestamps when you don't specify a props.conf?

2014-07-23 02:44:46,913 INFO [com.mpi.saml.plugin.SamlTokenRepositoryCleanupTask] Deleting the expired tokens from SAML repository
2014-07-23 02:46:21,620 INFO [com.media3k.escene.util.background.SsoNonceCleaner] DEFAULT.SsoNonceCleaner executing at Wed Jul 23 02:46:21 EDT 2014
2014-07-23 02:59:46,914 INFO [com.mpi.saml.plugin.SamlTokenRepositoryCleanupTask] Deleting the expired tokens from SAML repository
2014-07-23 03:01:21,693 INFO [com.media3k.escene.util.background.SsoNonceCleaner] DEFAULT.SsoNonceCleaner executing at Wed Jul 23 03:01:21 EDT 2014
2014-07-23 03:06:21,621 INFO [com.media3k.escene.util.background.SsoNonceCleaner] DEFAULT.SsoNonceCleaner executing at Wed Jul 23 03:06:21 EDT 2014
2014-07-23 03:14:15,379 ERROR [com.media3k.escene.util.background.TempFileCleaner] Unable to find ingesting Programs to activate.
java.lang.NullPointerException: null
at com.qumu.prg.builder.ProgramAssociationService.findProgramContentAssociation(ProgramAssociationService.java:168) ~[mpi-bcast.jar:na]
at com.media3k.escene.util.background.TempFileCleaner.processImage(TempFileCleaner.java:115) [adminportal.jar:na]
at com.media3k.escene.util.background.TempFileCleaner.access$200(TempFileCleaner.java:39) [adminportal.jar:na]
at com.media3k.escene.util.background.TempFileCleaner$1.call(TempFileCleaner.java:70) [adminportal.jar:na]
at com.media3k.escene.util.background.TempFileCleaner$1.call(TempFileCleaner.java:57) [adminportal.jar:na]
at com.mpi.framework.threadcontext.ThreadContexts$ContextCallableImpl.call(ThreadContexts.java:141) [foundation.jar:na]
at com.media3k.escene.util.background.TempFileCleaner.finalizeEgcPrograms(TempFileCleaner.java:57) [adminportal.jar:na]
at com.media3k.escene.util.background.TempFileCleaner.executeInternal(TempFileCleaner.java:51) [adminportal.jar:na]
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:113) [spring-context-support.jar:3.2.1.RELEASE]
at org.quartz.core.JobRunShell.run(JobRunShell.java:216) [quartz.jar:na]
at com.media3k.escene.util.background.TaskExecutionServiceThreadPool$QuartzTask.call(TaskExecutionServiceThreadPool.java:93) [mpi-lodgepole.jar:na]
at com.media3k.escene.util.background.TaskExecutionServiceThreadPool$QuartzTask.call(TaskExecutionServiceThreadPool.java:82) [mpi-lodgepole.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_65]
at com.mpi.framework.task.impl.juc.JUCThreadPool$InternalFutureTask.run(JUCThreadPool.java:328) [foundation.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
2014-07-23 03:14:15,379 INFO [com.media3k.escene.util.background.TempFileCleaner] CentralConfig.filerTempDir not set or does not exist; using D:\Temp for file upload directory.
2014-07-23 03:14:15,380 INFO [com.media3k.escene.util.background.TempFileCleaner] Cleaning up the temp folder: D:\Temp
2014-07-23 03:14:46,915 INFO [com.mpi.saml.plugin.SamlTokenRepositoryCleanupTask] Deleting the expired tokens from SAML repository
2014-07-23 03:21:21,634 INFO [com.media3k.escene.util.background.SsoNonceCleaner] DEFAULT.SsoNonceCleaner executing at Wed Jul 23 03:21:21 EDT 2014

[linu1988]

Hello,
The below would actually be appropriate in your case.

NO_BINARY_CHECK = 1
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TRUNCATE = 0
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = true

OR

NO_BINARY_CHECK = 1
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TRUNCATE = 0
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE =\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}

Thanks,
L

[richgalloway]

Your second LINE_BREAKER string should do it. If not, try '([\r\n]+)(\d+)-(\d+)-(\d+)'

[a212830]

Oh, maybe...?

MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = 1
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TIME_PREFIX = ^
TRUNCATE = 999999
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}