Getting Data In
Highlighted

props.conf help

Champion

Hi,

I have a multi-line feed that I'm having problems with - hoping someone can help me. Here's my props.conf:

MAXTIMESTAMPLOOKAHEAD = 30
NOBINARYCHECK = 1
TIMEFORMAT = %Y-%m-%d %H:%M:%S,%3N
TIME
PREFIX = ^
TRUNCATE = 999999
SHOULDLINEMERGE = false
LINE
BREAKER = ([\r\n]+)\d{4}\s\w{2}\s\d{2}

And, here's some sample data. The break is not occurring on the lines properly. Pretty sure my LINE_BREAKER is wrong. On a side note, if I just let splunk do the work, everything is wonderful, but this is not how we want to eng our systems. Is there any way to tell what splunk is applying for timestamps when you don't specify a props.conf?

2014-07-23 02:44:46,913 INFO [com.mpi.saml.plugin.SamlTokenRepositoryCleanupTask] Deleting the expired tokens from SAML repository
2014-07-23 02:46:21,620 INFO [com.media3k.escene.util.background.SsoNonceCleaner] DEFAULT.SsoNonceCleaner executing at Wed Jul 23 02:46:21 EDT 2014
2014-07-23 02:59:46,914 INFO [com.mpi.saml.plugin.SamlTokenRepositoryCleanupTask] Deleting the expired tokens from SAML repository
2014-07-23 03:01:21,693 INFO [com.media3k.escene.util.background.SsoNonceCleaner] DEFAULT.SsoNonceCleaner executing at Wed Jul 23 03:01:21 EDT 2014
2014-07-23 03:06:21,621 INFO [com.media3k.escene.util.background.SsoNonceCleaner] DEFAULT.SsoNonceCleaner executing at Wed Jul 23 03:06:21 EDT 2014
2014-07-23 03:14:15,379 ERROR [com.media3k.escene.util.background.TempFileCleaner] Unable to find ingesting Programs to activate.
java.lang.NullPointerException: null
at com.qumu.prg.builder.ProgramAssociationService.findProgramContentAssociation(ProgramAssociationService.java:168) ~[mpi-bcast.jar:na]
at com.media3k.escene.util.background.TempFileCleaner.processImage(TempFileCleaner.java:115) [adminportal.jar:na]
at com.media3k.escene.util.background.TempFileCleaner.access$200(TempFileCleaner.java:39) [adminportal.jar:na]
at com.media3k.escene.util.background.TempFileCleaner$1.call(TempFileCleaner.java:70) [adminportal.jar:na]
at com.media3k.escene.util.background.TempFileCleaner$1.call(TempFileCleaner.java:57) [adminportal.jar:na]
at com.mpi.framework.threadcontext.ThreadContexts$ContextCallableImpl.call(ThreadContexts.java:141) [foundation.jar:na]
at com.media3k.escene.util.background.TempFileCleaner.finalizeEgcPrograms(TempFileCleaner.java:57) [adminportal.jar:na]
at com.media3k.escene.util.background.TempFileCleaner.executeInternal(TempFileCleaner.java:51) [adminportal.jar:na]
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:113) [spring-context-support.jar:3.2.1.RELEASE]
at org.quartz.core.JobRunShell.run(JobRunShell.java:216) [quartz.jar:na]
at com.media3k.escene.util.background.TaskExecutionServiceThreadPool$QuartzTask.call(TaskExecutionServiceThreadPool.java:93) [mpi-lodgepole.jar:na]
at com.media3k.escene.util.background.TaskExecutionServiceThreadPool$QuartzTask.call(TaskExecutionServiceThreadPool.java:82) [mpi-lodgepole.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.065]
at com.mpi.framework.task.impl.juc.JUCThreadPool$InternalFutureTask.run(JUCThreadPool.java:328) [foundation.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0
65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.065]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0
65]
2014-07-23 03:14:15,379 INFO [com.media3k.escene.util.background.TempFileCleaner] CentralConfig.filerTempDir not set or does not exist; using D:\Temp for file upload directory.
2014-07-23 03:14:15,380 INFO [com.media3k.escene.util.background.TempFileCleaner] Cleaning up the temp folder: D:\Temp
2014-07-23 03:14:46,915 INFO [com.mpi.saml.plugin.SamlTokenRepositoryCleanupTask] Deleting the expired tokens from SAML repository
2014-07-23 03:21:21,634 INFO [com.media3k.escene.util.background.SsoNonceCleaner] DEFAULT.SsoNonceCleaner executing at Wed Jul 23 03:21:21 EDT 2014

Tags (2)
0 Karma
Highlighted

Re: props.conf help

Champion

Oh, maybe...?

MAXTIMESTAMPLOOKAHEAD = 30
NOBINARYCHECK = 1
TIMEFORMAT = %Y-%m-%d %H:%M:%S,%3N
TIME
PREFIX = ^
TRUNCATE = 999999
SHOULDLINEMERGE = false
LINE
BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}

0 Karma
Highlighted

Re: props.conf help

SplunkTrust
SplunkTrust

Your second LINE_BREAKER string should do it. If not, try '([\r\n]+)(\d+)-(\d+)-(\d+)'

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: props.conf help

Champion

Hello,
The below would actually be appropriate in your case.

NO_BINARY_CHECK = 1
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TRUNCATE = 0
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = true

OR

NO_BINARY_CHECK = 1
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TRUNCATE = 0
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE =\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}

Thanks,
L

0 Karma