topic Re: Parsing in Splunk Search

Parsing

user3344 — Tue, 28 Jan 2025 14:24:44 GMT

Hello community,

I need help with configuring Splunk to correctly process timestamp information in my UDP messages. When I send messages starting with a pattern like <\d+>, for example:

<777> 2025-01-03T06:12:19.236514-08:00 hello world

Splunk substitutes the original timestamp with the current date and local host address. Consequently, what I see in Splunk is:

Jan 28 14:27:25 127.0.0.1 2025-01-03T06:12:19.236514-08:00 hello world

I would like to know how to disable this behavior so that the actual timestamp from the message is preserved in the event. I have attempted to configure TIME_FORMAT and TIME_PREFIX in the props.conf file, but it seems those settings are applied after Splunk substitutes the timestamp with the current date and local host.

As a workaround, I implemented the following in props.conf:

[my_sourcetype] EXTRACT-HostName = \b(?P\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+(-\d{2}:\d{2})?) EVAL-_time = strptime(extracted_time, "%Y-%m-%dT%H:%M:%S.%6N%z")

Is there a better way to achieve this? Any guidance would be greatly appreciated!

Thank you!

Re: Parsing

gcusello — Tue, 28 Jan 2025 14:45:24 GMT

Hi @user3344 ,

i suppose that you're speaking of Splunk as syslog receiver.

I tried in many ways to avoid to add this prefix without any result, every syslog receiver adds at the beginning of the event the timestamp and the sender ip.

It isn't a problem from Splunk, if you use rsyslog as receiver, you have the same behaviour.

The only way is, if it's wrong, to modify the timestamp format to take the second one and not the one added by the syslog receiver.

You could try to remove the header using SEDCMD command in props.conf, but anyway, you have to configure the second timestamp as the correct timestamp.

Ciao.

Giuseppe

Re: Parsing

richgalloway — Tue, 28 Jan 2025 15:20:08 GMT

First, one should not be receiving UDP (or TCP) directly into a Splunk instance. Instead, use a dedicated syslog receiver (such as syslog-ng) to save the data to disk and monitor the files with a Universal Forwarder.

If that's not feasible, try these props to better extract timestamps from the events.

[my_sourcetype] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+)\< TIME_PREFIX = \>\s TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%z MAX_TIMESTAMP_LOOKAHEAD = 32 EXTRACT-HostName = \b(?P\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+(-\d{2}:\d{2})?)

Re: Parsing

user3344 — Tue, 28 Jan 2025 18:48:44 GMT

That doesn't help. _time still represents current time, not time in the event.

Re: Parsing

user3344 — Tue, 28 Jan 2025 18:51:53 GMT

Could you explain a bit more what do you mean by

@gcusello wrote:
The only way is, if it's wrong, to modify the timestamp format to take the second one and not the one added by the syslog receiver.

?

is approach with having that

EXTRACT-extracted_time = \b(?P<extracted_time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+(-\d{2}:\d{2})?) EVAL-_time = strptime(extracted_time, "%Y-%m-%dT%H:%M:%S.%6N%z")

in props.conf right?

Re: Parsing

isoutamo — Tue, 28 Jan 2025 19:27:15 GMT

You must put those configurations in first full splunk enterprise instance from source to indexers. If you have separate HF which is endpoint for those udp feed, then those must be there. If that is UF and then you are send events through IHF then add those conf there. And if there haven’t been any HF before indexers then add those configurations in all indexers.
And as said it’s better to use real syslog server to terminate syslog feeds a use e.g. a UF to collects events from files or use SC4S for that.

Re: Parsing

PickleRick — Wed, 29 Jan 2025 10:07:36 GMT

Normally timestamp extraction (given it's properly configured) works pretty well and Splunk doesn't have to resort to the fallback action of assigning the current timestamp to the event.

But the data ingestion must be properly configured - the sourcetype must have correct settings for the given time format and the sourcetype must be defined at the right place (on the correct component).

In your case we have no idea what the ingestion process looks like, how many components are involved, where the settings are defined, what sourcetypes you are using and so on.

Furthermore the fact that

<777> 2025-01-03T06:12:19.236514-08:00 hello world

event which looks like "almost normal" syslog message (the <777> is definitely not a correct facility/priority combination) is getting transformed into

Jan 28 14:27:25 127.0.0.1 2025-01-03T06:12:19.236514-08:00 hello world

suggests that there is some intermediate step (the 127.0.0.1 part is not a part of the original message).