Getting Data In

How to extract Date and time logged to different locations in the same file?

NickyCarlisle
Engager

Hello, I have a log file where the date is at the top of the log and the time for each event is at the start of each line, so something like this:

-- Log Continued 03/28/2022 00:00:00.471 --
00:00:36.526 xxxxx
00:04:01.809 xxxxx
00:04:09.267 xxxxx
00:10:19.039 xxxxx

How would I extract the date/ time using props.conf or similar?

0 Karma
1 Solution

PickleRick
Ultra Champion

OK, that was a bit of a long shot since I've never done it myself but.

1) The time parsing comes as one of the first stages of event processing during ingestion

2) You can manipulate metadata with transforms

3) You can use ingest-time evals to modify values not only with simple regexes but also with quite a significant range of normal splunk operations.

See the https://conf.splunk.com/files/2020/slides/PLA1154C.pdf for example.

View solution in original post

VatsalJagani
Champion

You cannot extract the date from the header of the file but I think if you date in the file name then you should be able to extract the date from the file name and time from the event itself. This should work as far as you have the proper date format as part of your file/source name.

VatsalJagani_0-1649226282336.png

Use below for props.conf for time extraction.

 

TIME_FORMAT = %T.%3N
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 20

 

Please read - https://docs.splunk.com/Documentation/Splunk/8.2.6/Data/HowSplunkextractstimestamps 

 

I hope this resolves your problem. If so please consider upvoting!!!

PickleRick
Ultra Champion

If you have the date in source field, you can as well use ingest-time eval and metadata manipulation to modify the _time field explicitly.

VatsalJagani
Champion

@PickleRick - 😊That is new to me that you can change timestamp as well with metadata manipulation.
Can you please share any document or config example for that? I would really appreciate that.

0 Karma

PickleRick
Ultra Champion

OK, that was a bit of a long shot since I've never done it myself but.

1) The time parsing comes as one of the first stages of event processing during ingestion

2) You can manipulate metadata with transforms

3) You can use ingest-time evals to modify values not only with simple regexes but also with quite a significant range of normal splunk operations.

See the https://conf.splunk.com/files/2020/slides/PLA1154C.pdf for example.

PickleRick
Ultra Champion

If you want to split the file into separate event per each line and use the date from the file's header... then you can't do it by splunk alone. You can't "parse and then split" events. You can't "carry" some information from one event over to another one. Each event is parsed and interpreted separately. You'd need to have some script which would prepare your file by cutting and pasting the date into each line and only ingest such prepared file, not the original one.

0 Karma
Get Updates on the Splunk Community!

Set Up More Secure Configurations in Splunk Enterprise With Config Assist

This blog post is part 3 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...