Getting Data In

Raw import not getting the full data. Is this a CR/LF issue that can be fixed with a configuration file edit?

ipicbc
Explorer

I am not getting the full event on ingestion from a log file. I am assuming it's a CR/LF problem that would be fixed by a config file tweak.

This is some data :

2017-04-18 04:00:53,373 [6968] INFO  BatchLog - Will execute jobName: QP_EODSnapshotBase with JobTask: DailyTradeSnapshotJobTask.

2017-04-18 04:02:55,224 [6968] INFO  BatchLog - Job Name:   QP_EODSnapshotBase
JobTask:    DailyTradeSnapshotJobTask
Start Time: 18/04/2017 04:00
End Time:   18/04/2017 04:02


2017-04-18 04:02:55,240 [6968] INFO  BatchLog - Exclusion list retrieved for jobName QP_EODSnapshotEnergy : 

Each line ends with a CR/LF. The first and last records in the snippet import OK, the middle one does not bring in the lines with Start Time and End Time.

Here is the props.conf entry for this sourcetype:

[qpbatch]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
BREAK_ONLY_BEFORE_DATE = True

Perhaps I am breaking the break_only_before_date logic because there is a date on those lines?!?!?!?

Any thoughts as to why this happens?

Thanks

1 Solution

gvmorley
Contributor

Hi,

Have a go with this for your props.conf

[qpbatch]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q
MAX_TIMESTAMP_LOOKAHEAD = 23
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_TIME = True
TRUNCATE = 10000

This way you're telling Splunk that the timestamp starts at the beginning of lines and the exact format to look for.

I did a quick test and got this:

alt text

See if that helps you get closer to a solution.

View solution in original post

gvmorley
Contributor

Hi,

Have a go with this for your props.conf

[qpbatch]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q
MAX_TIMESTAMP_LOOKAHEAD = 23
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_TIME = True
TRUNCATE = 10000

This way you're telling Splunk that the timestamp starts at the beginning of lines and the exact format to look for.

I did a quick test and got this:

alt text

See if that helps you get closer to a solution.

ipicbc
Explorer

I will give that a try. I tried

BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d\s*[[]\d*[]]

and that seemed to work.

Many thanks

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...