topic Re: Extracting ISO8601 timestamp in Getting Data In

Extracting ISO8601 timestamp

mbrownoutside — Wed, 12 Feb 2020 02:34:28 GMT

Hello,

I’m working on a powershell inputs and am stuck in regards to extracting the timestamp.

An event is stdout from my script as follows:

2020-02-05T14:11:36.000000-05:00 actinguser_userid="WJ" affecteduser_userid="DG" affecteduser_name="G,D" actiondescription="Password reset by administrator. "

I am using the following props:

[this:adminevents]
SHOULD_LINEMERGE = false
CHECK_FOR_HEADER = false
#KV_MODE = auto
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N
#TIME_PREFIX = Timestamp\s*:\s
TZ = -05:00

Is it possible to extract the timezone directly by parsing the timestamp?

This is my first run through of an extraction, so I apologize if it's simple.

Also, how do I debug extraction? Is there a way to enable debugging so that a specific sourcetype's extraction steps are logged to _internal?

Thanks,

Matt

Re: Extracting ISO8601 timestamp

manjunathmeti — Wed, 30 Sep 2020 04:10:28 GMT

hi @mbrownoutside, Your event has timezone (-05:00), Splunk will automatically extract and use it. TIME_FORMAT starts reading after the TIME_PREFIX. Here there is no TIME_PREFIX. Remove/comment TIME_FORMAT and TZ and check.

Re: Extracting ISO8601 timestamp

to4kawa — Wed, 12 Feb 2020 14:03:43 GMT

I test your sample log.
my props.conf does not have time related settings.
but time is extracted property.

 [this:adminevents]
 SHOULD_LINEMERGE = false
 CHECK_FOR_HEADER = false

At _time, from -05:00 to +09:00. from local time to local time.
That is enough, maybe.

Re: Extracting ISO8601 timestamp

to4kawa — Wed, 12 Feb 2020 14:03:44 GMT

TIME_FORMAT = %FT%T.%6N%:z

cf time format

Re: Extracting ISO8601 timestamp

mbrownoutside — Wed, 12 Feb 2020 14:03:45 GMT

It appears that I am, in fact, not using ISO8601, but RFC3339.

This page goes into the differences and similarities. RFC 3339 is more strict, and has provisions for timezone.

This brought me to this answers post.

I expect the answer to be and will test now:

%Y-%m-%dT%H:%M:%S.%6N%z

But in the doc you linked, %z does not have a definition for -05:00, but only -0500 or -5:00 or -05:00:00. I could explicitly use %:::z:00, but I then believe splunk may not properly extract the timstamp.

Re: Extracting ISO8601 timestamp

mbrownoutside — Wed, 12 Feb 2020 14:03:46 GMT

This does not work. I want to state a few things here to be transparent. This is a legacy OS, and I had to install an older version of Splunk. I have pushed the new datetime.xml to the client, stating "Version 4.0", therefore, unless there were changes in the strftime() support from the version I am on, I don't expect there to be challenges.

Please note that I also used the TIME_FORMAT %Y-%m-%dT%H:%M:%S.%6N-05:00 without luck.

Re: Extracting ISO8601 timestamp

to4kawa — Wed, 12 Feb 2020 14:03:46 GMT

| makeresults 
| eval _raw="2020-02-05T14:11:36.000000-05:00 actinguser_userid=\"WJ\" affecteduser_userid=\"DG\" affecteduser_name=\"G,D\" actiondescription=\"Password reset by administrator. \""
| rex "(?<time>\S+)"
| eval _time=strptime(time,"%FT%T.%6N%:z")
| eval time2=strftime(_time,"%FT%T.%6N%:z")

@mbrownoutside no problem.

Re: Extracting ISO8601 timestamp

mbrownoutside — Wed, 12 Feb 2020 14:03:47 GMT

Thank you for providing a test. This works for any existing records at search time (of course).

I am performing extraction in props.conf within the TA on the local UF. That strptime() string does not work currently within the props.conf as:

[this:adminevents]
SHOULD_LINEMERGE = false
CHECK_FOR_HEADER = false
TIME_FORMAT = %FT%T.%6N%:z

Is it safe to assume that I also need to place this on my HF (or indexers)? I was expecting that props.conf operates on the UF to perform timestamp extraction?

Thanks,

Matt

Re: Extracting ISO8601 timestamp

mbrownoutside — Wed, 12 Feb 2020 14:30:06 GMT

I'm not sure where you're replies are, but I am seeing them via email notifications.

I have tested with the following props.conf on the UF only:

[this:adminevents]
SHOULD_LINEMERGE = false
CHECK_FOR_HEADER = false
#KV_MODE = auto
#TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N-05:00
#TIME_FORMAT = %FT%T.%6N%:z
#TIME_PREFIX = Timestamp\s*:\s
#TZ = -05:00

The _time is not extracted.