Getting Data In

Timestamp and Universal Forwarder confusion

PickleRick
SplunkTrust
SplunkTrust

I'm a bit lost. Every piece of info that I find on the web (as well as materials from the Splunk's own trainings) say that UF does only very limited input preparation (line breaking, metadata adjustment, character encoding) but no real parsing work.

Thus I'm confused to find in logs that, for example:

12-03-2021 15:50:44.906 +0100 WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (128) characters of event. Defaulting to timestamp of previous event (Fri Dec 3 15:50:36 2021). Context:[...]

That would mean that some timestamp parsing does take place.

But do I still need to put my timestamp extraction config on the HF? (I use UF -> HF-> idx)

How does it correspond to possible settings about breaking on timestamp? If I want to break (which should happen on UF, right?) on timestamp, do I need to provide timestamp format on both UF (for breaking) and HF (for parsing)?

Labels (2)
0 Karma

harsmarvania57
Ultra Champion

It depends, if you have sourcetype which is using INDEXED_EXTRACTIONS then those data goes through StructuredParsing pipeline which means datetime parsing happen on UF. See https://wiki.splunk.com/Community:HowIndexingWorks (4. Detail Diagram - UF/LWF to Indexer)

PickleRick
SplunkTrust
SplunkTrust

This UF is ingesting Exchange logs. If I remember correctly, they are structured but they don't use indexed extractions (but I might be wrong; I don't have this evironment at hand to check it quickly).

But in general, if I want to add another sourcetype, with no indexed extractions, I should normally add timestamp-related props on HF, right?

Thanks for the link!

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...

Modernize your Splunk Apps – Introducing Python 3.13 in Splunk

We are excited to announce that the upcoming releases of Splunk Enterprise 10.2.x and Splunk Cloud Platform ...

Step into “Hunt the Insider: An Splunk ES Premier Mystery” to catch a cybercriminal ...

After a whole week of being on call, you fell asleep on your keyboard, and you hit a sequence of buttons that ...