topic Re: Extracting Date and Time from Events in Getting Data In

Extracting Date and Time from Events

NealBr — Mon, 03 Aug 2020 15:20:22 GMT

How do I extract the date and time from my events?

Event Data Sample
-------------------------
Jun 4 01:27:01 rofsso504a Usage: /dev/sda1 16G 16G 20K 100% /
Jun 4 02:27:01 rofsso504a Usage: /dev/sda1 16G 16G 20K 100% /
Jun 4 00:27:01 rofsso504a Usage: /dev/sda1 16G 16G 20K 100% /
Jul 31 22:27:01 rofsso504a Usage: /dev/sda4 210G 53G 157G 26% /home
Jul 31 08:27:01 rofsso504a Usage: /dev/sda4 210G 53G 157G 26% /home

My Search
-----------------
index=sso host=rofsso504* PartitionDiskSpaceUsed>25 earliest=-2mon
| rename _raw as Event host as Host
| eval Timestamp=strftime(_time, "%b %d %H:%M:%S")
| table Host _time Timestamp PartitionDiskSpaceUsed Event
| sort Host -Timestamp | table _time Timestamp PartitionDiskSpaceUsed Event

What I want
------------------
I want the Timestamp column to contain the correct Event Date and Time, but currently it shows the DateTime of the search.

2020-06-04 00:50:56 Jun 04 01:27:01 100 Jun 4 01:27:01 rofsso504a Usage: /dev/sda1 16G 16G 20K 100% /
2020-06-04 00:50:56 Jun 04 02:27:01 100 Jun 4 02:27:01 rofsso504a Usage: /dev/sda1 16G 16G 20K 100% /
2020-06-04 00:50:56 Jun 04 00:27:01 100 Jun 4 00:27:01 rofsso504a Usage: /dev/sda1 16G 16G 20K 100% /
2020-07-31 00:50:56 Jul 31 22:27:01 26 Jul 31 22:27:01 rofsso504a Usage: /dev/sda4 210G 53G 157G 26% /home
2020-07-31 00:50:56 Jul 31 08:27:01 26 Jul 31 08:27:01 rofsso504a Usage: /dev/sda4 210G 53G 157G 26% /home

What I get
------------
2020-06-04 00:50:56 Jun 04 ~~00:50:56~~ 100 Jun 4 01:27:01 rofsso504a Usage: /dev/sda1 16G 16G 20K 100% /
2020-06-04 00:50:56 Jun 04 ~~00:50:56~~ 100 Jun 4 02:27:01 rofsso504a Usage: /dev/sda1 16G 16G 20K 100% /
2020-06-04 00:50:56 Jun 04 ~~00:50:56~~ 100 Jun 4 00:27:01 rofsso504a Usage: /dev/sda1 16G 16G 20K 100% /
2020-07-31 00:50:56 Jul 31 ~~00:50:56~~ 26 Jul 31 22:27:01 rofsso504a Usage: /dev/sda4 210G 53G 157G 26% /home
2020-07-31 00:50:56 Jul 31 ~~00:50:56~~ 26 Jul 31 08:27:01 rofsso504a Usage: /dev/sda4 210G 53G 157G 26% /home

Re: Extracting Date and Time from Events

richgalloway — Mon, 03 Aug 2020 16:55:36 GMT

It looks like your query does not match your output. I see two timestamp fields in the table command, but three timestamps are shown in the output.

Can you share the props.conf settings for this sourcetype? There may be an error in how the data is onboarded. Specifically. the timestamp may not be set correctly.

Re: Extracting Date and Time from Events

NealBr — Mon, 03 Aug 2020 17:52:57 GMT

Actually, the output has only two DateTime fields (the third one is just the event text). In total there are four fields. For example:

_time: 2020-06-04 00:50:56
Timestamp: Jun 04 00:50:56
PartitionDiskSpaceUsed: 100
Event: Jun 4 01:27:01 rofsso504a Usage: /dev/sda1 16G 16G 20K 100% /

Sorry, I'm new to Splunk and I don't understand your question: "Can you share the props.conf settings for this sourcetype?"

What is my sourcetype?
I've no idea where to find my props.config file (but I'm searching online for more info about it.)

Re: Extracting Date and Time from Events

NealBr — Mon, 03 Aug 2020 18:13:05 GMT

Found it!

CHARSET = AUTO

EVENT_BREAKER = ([\r\n]+)

EVENT_BREAKER_ENABLE = TRUE

LINE_BREAKER = ([\r\n]+)

MAX_TIMESTAMP_LOOKAHEAD = 100

SHOULD_LINEMERGE = 0

TIME_PREFIX = .+sso\d+.\s

TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N

TRUNCATE = 2500

#NO_BINARY_CHECK = true

category = Custom

disabled = false

Re: Extracting Date and Time from Events

richgalloway — Mon, 03 Aug 2020 18:24:30 GMT

Jun 4 01:27:01 rofsso504a Usage: /dev/sda1 16G 16G 20K 100% /

Assuming that's the correct sourcetype, then the timestamp settings don't match the data, which means Splunk will default to the current time. You should see messages in splunkd.log about not being able to find timestamps in the data. Use this search to find them.

index=_internal sourcetype=splunkd component=DateParserVerbose log_level=WARN

Try these props.conf settings to better parse those timestamps.

EVENT_BREAKER = ([\r\n]+) EVENT_BREAKER_ENABLE = TRUE LINE_BREAKER = ([\r\n]+) MAX_TIMESTAMP_LOOKAHEAD = 100 SHOULD_LINEMERGE = 0 TIME_PREFIX = ^ TIME_FORMAT = %b %d %H:%M:%S TRUNCATE = 2500

I've found the easiest way to test props.conf settings is to use the Add Data wizard. Save some sample events to a file on your workstation then go to Settings->Add Data. Select the option to upload a file and choose the file with the sample data. In the "Set Source Type" window, click on Advanced and enter the properties in the boxes. Click Apply to see if times are parsed correctly. If the settings work, cancel the upload and put the props.conf settings on your indexers.