Getting Data In

merging data at index time or using the second date for the time stamp


I have a log that looks like this:

2010/06/28 12:44:21 -

-ERROR(Version: 1.0 Buildguy from 2009-05-12 08.45.26) : 2010/06/28 12:44:21....

when I index it with the main index I get two events:

 2010/06/28 12:44:21        2010/06/28 12:44:21 -

 2009-05-12 12:44:21        -ERROR(Version: 1.0 Buildguy from 2009-05-12 08.45.26) :        
                             2010/06/28 12:44:21....

my problem here is the Date for the second event is using one from the error message not the second date which is the one I need for my timestamp.

I also tried a props.conf that looks like this:

 BREAK_ONLY_BEFORE = ^\d\d/\d\d/\d\d \d\d:\d\d:\d\d -

I only get one event using this I2 index:

 2010/06/28 12:44:21        2010/06/28 12:44:21 -

Is there any way I can index this merging the Date with the error message or pull the second date from the Error message for the timestamp. I was not sure If the space between the Date and ERROR message was the problem and if it is, is there a way around this so i can merge these together.

Tags (1)

Super Champion

I think your event merging problem would be solved with the following props.conf entry:

BREAK_ONLY_BEFORE = ^\d{4}/\d\d/\d\d \d\d:\d\d:\d\d -

The BREAK_ONLY_BEFORE_DATE is True by default (despite what the docs say), which can cause your issues here because multiple dates are present. (You could also potententially solve this by specifying TIME_FORMAT). Also notice that your BREAK_ONLY_BEFORE regex was incorrect. You have a 4 digit year, not a 2 digit one. The regex above should work.

If you want splunk to use your second timestamp you have a couple of options. However, the sample you provided hasn't given enough context to know for sure what comes before your second date. (Your first sample has 3 different dates, two of which are the same, and then). Does the 2nd date always occur after the word "from"? In your example the two dates are from different years, which also seems weird. (You can edit your question and add a few more lines which shoudl be enough to establish a pattern)

There are a couple of helpful docs on this topic too:

Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...