How do I specify a date+time from a log to be used for event breaking and time stamp purposes?

williamcharlton — Fri, 31 May 2019 19:15:09 GMT

Below is an example of a log I want to ingest. I want to end up with 8 events.
The timestamp for each event should be the date+time that begins in column 1 for each of the 8 items you see below. The event itself should be the blob of text that follows the date+time that begins in column 1 for each of the 8 items.

For the "2019-01-28 12:05:43 PM" event, the blob of text that belongs to this event starts with
"Submit response(" (i.e., line 2) and ends with ")" (i.e., line 11).

Note that the date in the XML element **** (i.e., 2019-01-11T12:06:52-05:00 on line 21) is NOT an event date. It's just part of that event's text blob.

I'm using the web Splunk>enterprise -> Add Data -> Upload dialog to test the ingestion.
My trouble is on the "Set Source Type" dialog that shows the preview of the list of "Time" and "Event" data:
Since the text blob for an event might include multiple lines of text, I'm using the Regex
"*\d{4}-\d{2}-\d{2}\s{1}\d{2}:\d{2}:\d{2}(\s{1}\w{2}|))"
to define the event break as being the date that begins each event.

The regex creates the events with the text blob as expected, but the "Time" value for each event is set to the date+time that I ingested the file, not the date+time from the event in the log.
By using my Regex to set to event breaks, it seems that Splunk ignores the date+time captured by the Regex.

How do I specify that the date+time that begins in column 1 for each of the 8 items- is the event break AND have Splunk use it for each event time stamp?

The log

2019-01-28 12:05:42 - Previous Log archived to D:\foo\5.txt
2019-01-28 12:05:43 PM - Submit response(
<?xml version="1.0" encoding="utf-16"?>
<Response xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Message>Success: Success: Foo Adapter Submission Plugin - Submission Successful.
Success: The foo bar was successfully submitted to the Foo system.
</Message>
  <Status>Success</Status>
  <Success>true</Success>
</Response>
)
2019-01-28 12:05:49 PM - UploadDocument(documentType=foo123, filename=test.pdf, fileType=Pdf, content=System.Byte[], uniqueId=282084589)
2019-01-28 12:05:49 PM - O.R. Service validated!
2019-01-28 12:05:49 PM - Chunking D:\foo\test.pdf
2019-01-28 12:05:49 PM - Compressing D:\foo\test.pdf.  Size: 73116
2019-01-28 12:05:49 PM - Compressed size: 66236
2019-01-28 12:05:49 PM - UploadDocument response(
<?xml version="1.0" encoding="utf-16"?>
<ResponseOfGuid xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Message>The document was successfully uploaded to the Foo Repository.</Message>
  <DateSubmitted>2019-01-11T12:06:52-05:00</DateSubmitted>  
  <Status>Success</Status>
  <Success>true</Success>
</ResponseOfGuid>
)

Re: How do I specify a date+time from a log to be used for event breaking and time stamp purposes?

aromanauskas — Wed, 30 Sep 2020 00:44:46 GMT

Add the following settings in props.conf should break the event for you and maintain the correct timestamp.

TIME_FORMAT = %Y-%m-%d %H:%M:%S %p
SHOULD_LINEMERGE = true

Re: How do I specify a date+time from a log to be used for event breaking and time stamp purposes?

williamcharlton — Tue, 04 Jun 2019 13:47:53 GMT

I'm using the web Splunk>enterprise -> Add Data -> Upload dialog (not manually creating .conf files, so I assume your advice is applicable to the Splunk>enterprise -> Add Data -> Upload dialog, correct?

I also assume you mean for me to STOP using the custom "event break" RegEx and instead use the Timestamp TIME_FORMAT setting?

topic Re: How do I specify a date+time from a log to be used for event breaking and time stamp purposes? in Getting Data In

How do I specify a date+time from a log to be used for event breaking and time stamp purposes?

Re: How do I specify a date+time from a log to be used for event breaking and time stamp purposes?

Re: How do I specify a date+time from a log to be used for event breaking and time stamp purposes?