Getting Data In

Raw import not getting the full data. Is this a CR/LF issue that can be fixed with a configuration file edit?

ipicbc
Explorer

I am not getting the full event on ingestion from a log file. I am assuming it's a CR/LF problem that would be fixed by a config file tweak.

This is some data :

2017-04-18 04:00:53,373 [6968] INFO  BatchLog - Will execute jobName: QP_EODSnapshotBase with JobTask: DailyTradeSnapshotJobTask.

2017-04-18 04:02:55,224 [6968] INFO  BatchLog - Job Name:   QP_EODSnapshotBase
JobTask:    DailyTradeSnapshotJobTask
Start Time: 18/04/2017 04:00
End Time:   18/04/2017 04:02


2017-04-18 04:02:55,240 [6968] INFO  BatchLog - Exclusion list retrieved for jobName QP_EODSnapshotEnergy : 

Each line ends with a CR/LF. The first and last records in the snippet import OK, the middle one does not bring in the lines with Start Time and End Time.

Here is the props.conf entry for this sourcetype:

[qpbatch]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
BREAK_ONLY_BEFORE_DATE = True

Perhaps I am breaking the break_only_before_date logic because there is a date on those lines?!?!?!?

Any thoughts as to why this happens?

Thanks

1 Solution

gvmorley
Contributor

Hi,

Have a go with this for your props.conf

[qpbatch]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q
MAX_TIMESTAMP_LOOKAHEAD = 23
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_TIME = True
TRUNCATE = 10000

This way you're telling Splunk that the timestamp starts at the beginning of lines and the exact format to look for.

I did a quick test and got this:

alt text

See if that helps you get closer to a solution.

View solution in original post

gvmorley
Contributor

Hi,

Have a go with this for your props.conf

[qpbatch]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q
MAX_TIMESTAMP_LOOKAHEAD = 23
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_TIME = True
TRUNCATE = 10000

This way you're telling Splunk that the timestamp starts at the beginning of lines and the exact format to look for.

I did a quick test and got this:

alt text

See if that helps you get closer to a solution.

ipicbc
Explorer

I will give that a try. I tried

BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d\s*[[]\d*[]]

and that seemed to work.

Many thanks

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...