topic Re: Configure event timestamp recognition from epoch in Getting Data In

Configure event timestamp recognition from epoch

tommasoscarpa1 — Tue, 30 Jan 2024 15:04:27 GMT

Hello,

I have events in this format:

These events comes from HEC to an heavy forwarder and are then forwarded to indexers. I would like to set Splunk to recognize <epochtime> as the event timestamp.

<servername> and <metricname> are alphanumerical words with no whitespaces inside, while <metricvalue> is numerical. <epochtime> is a 10 digits, integer epoch time.

I've set up props.conf file on heavy forwarder as follows:

[sourcetypename]

TIME_FORMAT = %s

But events are not indexed with the correct timestamp. I also tried to add this property:
TIME_PREFIX = \S+\s\S+\s

But no luck.

Can you help me understand what am I doing wrong?

EDIT----

Log example:

mywebserver123 SOME_METRIC 1706569460 5

myotherwebserver456.domain.com ANY_OTHER_NAME 1706569582 3

Re: Configure event timestamp recognition from epoch

gcusello — Tue, 30 Jan 2024 14:47:42 GMT

Hi @tommasoscarpa1,

could you share a sample of your logs?

Ciao.

Giuseppe

Re: Configure event timestamp recognition from epoch

tommasoscarpa1 — Tue, 30 Jan 2024 14:53:04 GMT

Hi Giuseppe. I've added an example in my original post

Re: Configure event timestamp recognition from epoch

gcusello — Tue, 30 Jan 2024 14:56:27 GMT

Hi @tommasoscarpa1,

sorry I didn't see it!

Anyway, if before the timestamp in epochtime you always have METRIC_NAME , you could try

TIME_PREFIX = METRIC_NAME\s+ TIME_FORMAT = %s

Ciao.

Giuseppe

Re: Configure event timestamp recognition from epoch

tommasoscarpa1 — Tue, 30 Jan 2024 15:01:05 GMT

Hi @gcusello

No, that is just an example. I don't actually have the characters "METRIC_NAME" in my data. METRIC_NAME can be anything from "CPU usage" to "paychek value" or whatever.

That's why I tried TIME_PREFIX = \S+\s\S+\s at first.

I'll edit the examples in my original post to avoid misunderstandings

Re: Configure event timestamp recognition from epoch

gcusello — Tue, 30 Jan 2024 16:25:58 GMT

Hi @tommasoscarpa1 ,

if before the timestamp in epochtime you always have some char but not digits, you could try:

TIME_PREFIX = [a-zA-Z]\s+

this permits to be sure that Splunk uses the epochtime timestamp.

Ciao.

Giuseppe

Re: Configure event timestamp recognition from epoch

tommasoscarpa1 — Wed, 31 Jan 2024 08:46:05 GMT

--- IMPORTANT EDIT ---

After I accepted this solution, user @PickleRick suggested a way better one, so I am reporting it here for future use:

The /event endpoint gives you more flexibility than /raw so I'd advise to use /event anyway. But in order for HEC input _not_ to skip the timestamp recognition (which it does by default - it either gets the timestamp from the field pushed with (not in!) an event or assigns current timestamp), you must add the ?auto_extract_timestamp=true parameter to the url. Like

https://your_indexer:8088/services/collector/event?auto_extract_timestamp=true

Here below my original answer:

Hi @gcusello

I tried this too but no luck.

Eventually I solved my problem by changing the HEC endpoint.

I was sending data to "/services/collector/event" endpoint. I changed to "/services/collector/raw" and time was indexed correctly with only the TIME_FORMAT property.

Thank you for your help anyway!