topic Re: Raw import not getting the full data. Is this a CR/LF issue that can be fixed with a configuration file edit? in Getting Data In

Raw import not getting the full data. Is this a CR/LF issue that can be fixed with a configuration file edit?

ipicbc — Tue, 29 Sep 2020 13:44:48 GMT

I am not getting the full event on ingestion from a log file. I am assuming it's a CR/LF problem that would be fixed by a config file tweak.

This is some data :

2017-04-18 04:00:53,373 [6968] INFO  BatchLog - Will execute jobName: QP_EODSnapshotBase with JobTask: DailyTradeSnapshotJobTask.

2017-04-18 04:02:55,224 [6968] INFO  BatchLog - Job Name:   QP_EODSnapshotBase
JobTask:    DailyTradeSnapshotJobTask
Start Time: 18/04/2017 04:00
End Time:   18/04/2017 04:02


2017-04-18 04:02:55,240 [6968] INFO  BatchLog - Exclusion list retrieved for jobName QP_EODSnapshotEnergy :

Each line ends with a CR/LF. The first and last records in the snippet import OK, the middle one does not bring in the lines with Start Time and End Time.

Here is the props.conf entry for this sourcetype:

[qpbatch]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
BREAK_ONLY_BEFORE_DATE = True

Perhaps I am breaking the break_only_before_date logic because there is a date on those lines?!?!?!?

Any thoughts as to why this happens?

Thanks

Re: Raw import not getting the full data. Is this a CR/LF issue that can be fixed with a configuration file edit?

gvmorley — Tue, 18 Apr 2017 19:37:13 GMT

Hi,

Have a go with this for your props.conf

[qpbatch]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q
MAX_TIMESTAMP_LOOKAHEAD = 23
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_TIME = True
TRUNCATE = 10000

This way you're telling Splunk that the timestamp starts at the beginning of lines and the exact format to look for.

I did a quick test and got this:

See if that helps you get closer to a solution.

Re: Raw import not getting the full data. Is this a CR/LF issue that can be fixed with a configuration file edit?

ipicbc — Tue, 29 Sep 2020 13:45:06 GMT

I will give that a try. I tried

BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d\s*[[]\d*[]]

and that seemed to work.

Many thanks