Getting Data In

Delay Ingestion at the Universal Forwarder Until Event Is Complete

Path Finder

I have an interesting dilemma and I believe there is a solution, but I can use some advice on this one.

We have a log file that records submitted requests in the following format:

8038$$DRY ETCH$$3/9/2021 9:45:22 AM$$[More columns separated by double-$]

The first "field" is the request ID, then the "$$", then an area, then "$$", then the actual date/time, etc.

The issue is that every time a new entry is filled out, the next request ID is also added to the log file, so if the record above was the last one entered, the log file would end with the following records:

8037$$CMP$$3/9/2021 7:32:04 AM$$[More columns separated by double-$]
8038$$DRY ETCH$$3/9/2021 9:45:22 AM$$[More columns separated by double-$]

My problem is that at 9:45:22 AM, Splunk is ingesting this as the event:
"$$DRY ETCH$$3/9/2021 9:45:22 AM$$[More columns separated by double-$]

It ingests the request ID for the current event in the last event.  There are often hours between requests.  I want the ingestion to break immediately at the [\r\n] and NOT ingest the record ID from the last row in the log file until hours later when the event is completed as a new request gets entered for the request ID.

This is my props.conf stanza:
TIME_PREFIX = ^\s*\d+\${2}[\s\w\d-,]+\${2}
TIME_FORMAT = %m/%d/%Y %I:%M:%S %p
LINE_BREAKER = ([\r\n]+)^\s*\d+\${2}[\s\w\d-]+\${2}\d{1,2}\/\d{1,2}\/\d{4}\s+\d{1,2}:\d{2}:\d{2}\s+(AM|PM)\${2}
TRUNCATE = 999999

I was thinking about removing the LINE_BREAKER and adding the BREAK_ONLY_BEFORE = \d+\${2}[\s\w\d-]+\${2}\d{1,2}\/\d{1,2}\/\d{4}\s+\d{1,2}:\d{2}:\d{2}\s+(AM|PM)\${2}

Suggestions on the best method?  If I used BREAK_ONLY_BEFORE, will it still add the future request ID as the tail of the latest event?

If I use MUST_BREAK_AFTER = \$\$(No|Yes).*##(No|Yes)##(No|Yes)[\r\n]+

Would it still record the 4 digit number of the next request ID as a record by itself?

If I setup a transforms to throw out a 4 digit number that is the only thing in the record, would the universal forwarder send the 4 digit number the next time (doubt it because the UF keeps track of its last chunked position that it sent and the Heavy Forwarder is what throws out the request ID that came through - the UF doesn't even know it was thrown out)?  I'm stuck.  "Help me, Obi-Splunk Kenobi. You're my only hope."

0 Karma

Path Finder

Going to try and use:


I just think the request ID will still get ingested at the time it is logged and not later when the request ID event is completed/logged.

0 Karma

Path Finder

I found this post useful by @joesrepsolc, but it mostly covers entire file ingestion into a single event and was answered nicely by @bandit

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...