topic Re: How to make the _time from the source path? in Dashboards & Visualizations

How to make the _time from the source path?

power12 — Thu, 16 May 2024 18:42:03 GMT

I have the following source path which I am monitoring..I want splunk to use the _time from the source file.

/a/host*/ads/stdout.230621-125941

Thanks in Advance

Re: How to make the _time from the source path?

isoutamo — Tue, 27 Jun 2023 14:40:32 GMT

maybe these helps you?

r. Ismo

Re: How to make the _time from the source path?

power12 — Tue, 27 Jun 2023 15:05:23 GMT

@isoutamo Thank you so much for you reply. I tried the below but I am not sure what mistake I am doing

| makeresults | eval source="/logs/abc*/mods/stdout.230626-203719" | fields - _time ``` above set test data ``` | eval _time = strptime(replace(source, ".*\/stdout.(\d{6}\-\d{6}).*","\1") + tostring(random() % 86400,"duration"),"%Y%m%d%H:%M:%S")

Re: How to make the _time from the source path?

isoutamo — Tue, 27 Jun 2023 15:09:57 GMT

Your replace seems to be incorrect. It should be something like

| eval _time = strptime(replace(source, ".*/stdout.(\d{6}-\d{6}).*","20\1"),"%y%m%d-%H%M%S")

Re: How to make the _time from the source path?

power12 — Tue, 27 Jun 2023 16:12:07 GMT

@isoutamo This is what I used and I dont see _time field

| makeresults | eval source="/svc-logs/hostabc/mods/stdout.230626-203719" | fields - _time ``` above set test data ``` | eval _time = strptime(replace(source, ".*/stdout.(\d{6}-\d{6}).*","20\1"),"%y%m%d-%H%M%S")

Re: How to make the _time from the source path?

isoutamo — Tue, 27 Jun 2023 16:51:16 GMT

This _time is on your Time field on this event. Actually you should remove “| fields - _time” if you didn’t see Time on this one event.
Above is just test to get correct version for INGEST_EVAL expression to transforms.conf.

Re: How to make the _time from the source path?

power12 — Tue, 27 Jun 2023 16:56:43 GMT

I get the below

| makeresults
| eval source="/svc-logs/hostabc/mods/stdout.230626-203719"

``` above set test data ```
| eval _time = strptime(replace(source, ".*/stdout.(\d{6}-\d{6}).*","20\1"),"%y%m%d-%H%M%S")

Re: How to make the _time from the source path?

isoutamo — Tue, 27 Jun 2023 18:20:39 GMT

As I expecting that century is 20xx, you should use %Y instead of %y on strptime function.

Re: How to make the _time from the source path?

power12 — Tue, 27 Jun 2023 19:10:18 GMT

Thank you it worked

Re: How to make the _time from the source path?

power12 — Wed, 28 Jun 2023 20:54:26 GMT

@isoutamo

I Added below conf...but somehow the data still did not take the source path time.Is there other steps I need to add
props.conf

[< sourcetype name>]
TRANSFORMS-set_time = set_time_from_source_path

Added below in transforms.conf

[set_time_from_file_path]
INGEST_EVAL = _time = strptime(replace(source, ".*/stdout.(\d{6}-\d{6}).*","20\1"),"%Y%m%d-%H%M%S")

Re: How to make the _time from the source path?

isoutamo — Wed, 28 Jun 2023 22:04:53 GMT

Was it typo or are those name different on your conf files?

Re: How to make the _time from the source path?

power12 — Thu, 29 Jun 2023 15:34:16 GMT

I tried this as well

[< sourcetype name>]
TRANSFORMS-set_time = set_time_from_file_path

Added below in transforms.conf

[set_time_from_file_path]
INGEST_EVAL = _time = strptime(replace(source, ".*/stdout.(\d{6}-\d{6}).*","20\1"),"%Y%m%d-%H%M%S")

Should we have the "set_time" extension after the TRANSFORMS

Re: How to make the _time from the source path?

isoutamo — Thu, 29 Jun 2023 15:43:42 GMT

That should works. Have you restarted splunk and then remember that this is applied only a new ingested events, not for those which are already indexed.

Re: How to make the _time from the source path?

power12 — Thu, 29 Jun 2023 17:08:02 GMT

Yes...I did restart splunk after applying that...I tried mimicking the same on my local.

I went to my local (C:\Program Files\Splunk\etc\apps\search\local) where I set the transforms to be

[set_time_from_source_path]
INGEST_EVAL = _time = strptime(replace(source, ".*/stdout.(\d{6}-\d{6}).*","20\1"),"%Y%m%d-%H%M%S")

and in the props.

[sourcetime]
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
disabled = false
pulldown_type = true
TRANSFORMS-set_time = set_time_from_source_path

Restarted>

Then from WEB UI--->ADD DATA -->SELECTED FILE (_svc-logs_hostbac_mods_stdout.230629-053900) -->selected sourcetype (sourcetime) to see the preview I get errors(attached) that its failed to parse time

Re: How to make the _time from the source path?

isoutamo — Thu, 29 Jun 2023 19:06:43 GMT

On your example the source was “foo/bar/stdout.230629-123456” and now you are trying to use source “foo.bar.stdout.230629-123456”. Obviously that doesn’t match our regex on replace function. You must fix that to match your real source path!

Re: How to make the _time from the source path?

power12 — Thu, 29 Jun 2023 19:33:21 GMT

In the regex its looking for .*/stdout so even if its _ or . ..it matches everything until stdout right

INGEST_EVAL = _time = strptime(replace(source, ".*/stdout.(\d{6}-\d{6}).*","20\1"),"%Y%m%d-%H%M%S")

Re: How to make the _time from the source path?

isoutamo — Thu, 29 Jun 2023 19:41:43 GMT

It match everything before “/stdout…..” and you have “_stdout….” which obviously doesn’t match that.
So you must remove that / from regexp, then it should work.
https://regex101.com/ Is good place to test those.

Re: How to make the _time from the source path?

power12 — Thu, 29 Jun 2023 20:09:05 GMT

I can try that but I am testing on my windows local splunk installed trying to do the same from

C:\Users\vv\OneDrive-xx\Pictures\svc-logs\hostabc\mods\stdout.230629-081216.txt for this I changed it to

[set_time_from_source_path]
INGEST_EVAL = _time = strptime(replace(source, ".*\\stdout.(\d{6}-\d{6}).*","20\1"),"%Y%m%d-%H%M%S")

This also did not work.

I am assuming something with "20\1"..how do we change that..Can you please let me know what should be changed for this case

Re: How to make the _time from the source path?

isoutamo — Thu, 29 Jun 2023 20:20:09 GMT

You need just change this

INGEST_EVAL = _time = strptime(replace(source, ".*stdout.(\d{6}-\d{6}).*","20\1"),"%Y%m%d-%H%M%S")

then it works both windows and Linux.

When you are escaping \, the amount of needed \ is depending where you are using it (props, rex, …). Good rule of thumb is start with one and adding those as many as needed (sometimes until 4-6 is needed).

Re: How to make the _time from the source path?

power12 — Thu, 29 Jun 2023 20:34:51 GMT

Ya even tried that but somehow it says "could not use regex to parse timestamp from source::source::C:\Users\vv\OneDrive-xx\Pictures\svc-logs\hostabc\mods\stdou."