I am not getting the full event on ingestion from a log file. I am assuming it's a CR/LF problem that would be fixed by a config file tweak.
This is some data :
2017-04-18 04:00:53,373 [6968] INFO BatchLog - Will execute jobName: QP_EODSnapshotBase with JobTask: DailyTradeSnapshotJobTask.
2017-04-18 04:02:55,224 [6968] INFO BatchLog - Job Name: QP_EODSnapshotBase
JobTask: DailyTradeSnapshotJobTask
Start Time: 18/04/2017 04:00
End Time: 18/04/2017 04:02
2017-04-18 04:02:55,240 [6968] INFO BatchLog - Exclusion list retrieved for jobName QP_EODSnapshotEnergy :
Each line ends with a CR/LF. The first and last records in the snippet import OK, the middle one does not bring in the lines with Start Time and End Time.
Here is the props.conf entry for this sourcetype:
[qpbatch]
DATETIME_CONFIG =
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
BREAK_ONLY_BEFORE_DATE = True
Perhaps I am breaking the break_only_before_date logic because there is a date on those lines?!?!?!?
Any thoughts as to why this happens?
Thanks
Hi,
Have a go with this for your props.conf
[qpbatch]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q
MAX_TIMESTAMP_LOOKAHEAD = 23
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_TIME = True
TRUNCATE = 10000
This way you're telling Splunk that the timestamp starts at the beginning of lines and the exact format to look for.
I did a quick test and got this:
See if that helps you get closer to a solution.
Hi,
Have a go with this for your props.conf
[qpbatch]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q
MAX_TIMESTAMP_LOOKAHEAD = 23
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_TIME = True
TRUNCATE = 10000
This way you're telling Splunk that the timestamp starts at the beginning of lines and the exact format to look for.
I did a quick test and got this:
See if that helps you get closer to a solution.
I will give that a try. I tried
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d\s*[[]\d*[]]
and that seemed to work.
Many thanks