Getting Data In

Extracting ISO8601 timestamp

mbrownoutside
Path Finder

Hello,

I’m working on a powershell inputs and am stuck in regards to extracting the timestamp.

An event is stdout from my script as follows:

2020-02-05T14:11:36.000000-05:00 actinguser_userid="WJ" affecteduser_userid="DG" affecteduser_name="G,D" actiondescription="Password reset by administrator. "

I am using the following props:

[this:adminevents]
SHOULD_LINEMERGE = false
CHECK_FOR_HEADER = false
#KV_MODE = auto
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N
#TIME_PREFIX = Timestamp\s*:\s
TZ = -05:00

Is it possible to extract the timezone directly by parsing the timestamp?

This is my first run through of an extraction, so I apologize if it's simple.

Also, how do I debug extraction? Is there a way to enable debugging so that a specific sourcetype's extraction steps are logged to _internal?

Thanks,

Matt

0 Karma

to4kawa
Ultra Champion

I test your sample log.
my props.conf does not have time related settings.
but time is extracted property.

 [this:adminevents]
 SHOULD_LINEMERGE = false
 CHECK_FOR_HEADER = false

At _time, from -05:00 to +09:00. from local time to local time.
That is enough, maybe.

my props_conf extracts

to4kawa
Ultra Champion
TIME_FORMAT = %FT%T.%6N%:z

cf time format

0 Karma

mbrownoutside
Path Finder

It appears that I am, in fact, not using ISO8601, but RFC3339.

This page goes into the differences and similarities. RFC 3339 is more strict, and has provisions for timezone.

This brought me to this answers post.

I expect the answer to be and will test now:

%Y-%m-%dT%H:%M:%S.%6N%z

But in the doc you linked, %z does not have a definition for -05:00, but only -0500 or -5:00 or -05:00:00. I could explicitly use %:::z:00, but I then believe splunk may not properly extract the timstamp.

0 Karma

to4kawa
Ultra Champion
| makeresults 
| eval _raw="2020-02-05T14:11:36.000000-05:00 actinguser_userid=\"WJ\" affecteduser_userid=\"DG\" affecteduser_name=\"G,D\" actiondescription=\"Password reset by administrator. \""
| rex "(?<time>\S+)"
| eval _time=strptime(time,"%FT%T.%6N%:z")
| eval time2=strftime(_time,"%FT%T.%6N%:z")

@mbrownoutside no problem.

mbrownoutside
Path Finder

I'm not sure where you're replies are, but I am seeing them via email notifications.

I have tested with the following props.conf on the UF only:

[this:adminevents]
SHOULD_LINEMERGE = false
CHECK_FOR_HEADER = false
#KV_MODE = auto
#TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N-05:00
#TIME_FORMAT = %FT%T.%6N%:z
#TIME_PREFIX = Timestamp\s*:\s
#TZ = -05:00

The _time is not extracted.

0 Karma

mbrownoutside
Path Finder

Thank you for providing a test. This works for any existing records at search time (of course).

I am performing extraction in props.conf within the TA on the local UF. That strptime() string does not work currently within the props.conf as:

[this:adminevents]
SHOULD_LINEMERGE = false
CHECK_FOR_HEADER = false
TIME_FORMAT = %FT%T.%6N%:z

Is it safe to assume that I also need to place this on my HF (or indexers)? I was expecting that props.conf operates on the UF to perform timestamp extraction?

Thanks,

Matt

0 Karma

mbrownoutside
Path Finder

This does not work. I want to state a few things here to be transparent. This is a legacy OS, and I had to install an older version of Splunk. I have pushed the new datetime.xml to the client, stating "Version 4.0", therefore, unless there were changes in the strftime() support from the version I am on, I don't expect there to be challenges.

Please note that I also used the TIME_FORMAT %Y-%m-%dT%H:%M:%S.%6N-05:00 without luck.

0 Karma

manjunathmeti
Champion

hi @mbrownoutside, Your event has timezone (-05:00), Splunk will automatically extract and use it. TIME_FORMAT starts reading after the TIME_PREFIX. Here there is no TIME_PREFIX. Remove/comment TIME_FORMAT and TZ and check.

0 Karma
Get Updates on the Splunk Community!

Monitoring MariaDB and MySQL

In a previous post, we explored monitoring PostgreSQL and general best practices around which metrics to ...

Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...