Getting Data In

How do I specify a date+time from a log to be used for event breaking and time stamp purposes?

williamcharlton
Path Finder

Below is an example of a log I want to ingest. I want to end up with 8 events.
The timestamp for each event should be the date+time that begins in column 1 for each of the 8 items you see below. The event itself should be the blob of text that follows the date+time that begins in column 1 for each of the 8 items.

For the "2019-01-28 12:05:43 PM" event, the blob of text that belongs to this event starts with
"Submit response(" (i.e., line 2) and ends with ")" (i.e., line 11).

Note that the date in the XML element **** (i.e., 2019-01-11T12:06:52-05:00 on line 21) is NOT an event date. It's just part of that event's text blob.

I'm using the web Splunk>enterprise -> Add Data -> Upload dialog to test the ingestion.
My trouble is on the "Set Source Type" dialog that shows the preview of the list of "Time" and "Event" data:
Since the text blob for an event might include multiple lines of text, I'm using the Regex
"*\d{4}-\d{2}-\d{2}\s{1}\d{2}:\d{2}:\d{2}(\s{1}\w{2}|))"
to define the event break as being the date that begins each event.

The regex creates the events with the text blob as expected, but the "Time" value for each event is set to the date+time that I ingested the file, not the date+time from the event in the log.
By using my Regex to set to event breaks, it seems that Splunk ignores the date+time captured by the Regex.

How do I specify that the date+time that begins in column 1 for each of the 8 items- is the event break AND have Splunk use it for each event time stamp?

The log

2019-01-28 12:05:42 - Previous Log archived to D:\foo\5.txt
2019-01-28 12:05:43 PM - Submit response(
<?xml version="1.0" encoding="utf-16"?>
<Response xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Message>Success: Success: Foo Adapter Submission Plugin - Submission Successful.
Success: The foo bar was successfully submitted to the Foo system.
</Message>
  <Status>Success</Status>
  <Success>true</Success>
</Response>
)
2019-01-28 12:05:49 PM - UploadDocument(documentType=foo123, filename=test.pdf, fileType=Pdf, content=System.Byte[], uniqueId=282084589)
2019-01-28 12:05:49 PM - O.R. Service validated!
2019-01-28 12:05:49 PM - Chunking D:\foo\test.pdf
2019-01-28 12:05:49 PM - Compressing D:\foo\test.pdf.  Size: 73116
2019-01-28 12:05:49 PM - Compressed size: 66236
2019-01-28 12:05:49 PM - UploadDocument response(
<?xml version="1.0" encoding="utf-16"?>
<ResponseOfGuid xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Message>The document was successfully uploaded to the Foo Repository.</Message>
  <DateSubmitted>2019-01-11T12:06:52-05:00</DateSubmitted>  
  <Status>Success</Status>
  <Success>true</Success>
</ResponseOfGuid>
)
0 Karma
1 Solution

aromanauskas
Path Finder

Add the following settings in props.conf should break the event for you and maintain the correct timestamp.

TIME_FORMAT = %Y-%m-%d %H:%M:%S %p
SHOULD_LINEMERGE = true

View solution in original post

0 Karma

aromanauskas
Path Finder

Add the following settings in props.conf should break the event for you and maintain the correct timestamp.

TIME_FORMAT = %Y-%m-%d %H:%M:%S %p
SHOULD_LINEMERGE = true

0 Karma

williamcharlton
Path Finder

I'm using the web Splunk>enterprise -> Add Data -> Upload dialog (not manually creating .conf files, so I assume your advice is applicable to the Splunk>enterprise -> Add Data -> Upload dialog, correct?

I also assume you mean for me to STOP using the custom "event break" RegEx and instead use the Timestamp TIME_FORMAT setting?

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...