Getting Data In

Raw import not getting the full data. Is this a CR/LF issue that can be fixed with a configuration file edit?

ipicbc
Explorer

I am not getting the full event on ingestion from a log file. I am assuming it's a CR/LF problem that would be fixed by a config file tweak.

This is some data :

2017-04-18 04:00:53,373 [6968] INFO  BatchLog - Will execute jobName: QP_EODSnapshotBase with JobTask: DailyTradeSnapshotJobTask.

2017-04-18 04:02:55,224 [6968] INFO  BatchLog - Job Name:   QP_EODSnapshotBase
JobTask:    DailyTradeSnapshotJobTask
Start Time: 18/04/2017 04:00
End Time:   18/04/2017 04:02


2017-04-18 04:02:55,240 [6968] INFO  BatchLog - Exclusion list retrieved for jobName QP_EODSnapshotEnergy : 

Each line ends with a CR/LF. The first and last records in the snippet import OK, the middle one does not bring in the lines with Start Time and End Time.

Here is the props.conf entry for this sourcetype:

[qpbatch]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
BREAK_ONLY_BEFORE_DATE = True

Perhaps I am breaking the break_only_before_date logic because there is a date on those lines?!?!?!?

Any thoughts as to why this happens?

Thanks

1 Solution

gvmorley
Contributor

Hi,

Have a go with this for your props.conf

[qpbatch]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q
MAX_TIMESTAMP_LOOKAHEAD = 23
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_TIME = True
TRUNCATE = 10000

This way you're telling Splunk that the timestamp starts at the beginning of lines and the exact format to look for.

I did a quick test and got this:

alt text

See if that helps you get closer to a solution.

View solution in original post

gvmorley
Contributor

Hi,

Have a go with this for your props.conf

[qpbatch]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q
MAX_TIMESTAMP_LOOKAHEAD = 23
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_TIME = True
TRUNCATE = 10000

This way you're telling Splunk that the timestamp starts at the beginning of lines and the exact format to look for.

I did a quick test and got this:

alt text

See if that helps you get closer to a solution.

ipicbc
Explorer

I will give that a try. I tried

BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d\s*[[]\d*[]]

and that seemed to work.

Many thanks

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...