How to extract Date and time logged to different locations in the same file?

[NickyCarlisle]

Hello, I have a log file where the date is at the top of the log and the time for each event is at the start of each line, so something like this:

-- Log Continued 03/28/2022 00:00:00.471 --
00:00:36.526 xxxxx
00:04:01.809 xxxxx
00:04:09.267 xxxxx
00:10:19.039 xxxxx

How would I extract the date/ time using props.conf or similar?

[VatsalJagani]

You cannot extract the date from the header of the file but I think if you date in the file name then you should be able to extract the date from the file name and time from the event itself. This should work as far as you have the proper date format as part of your file/source name.

Use below for props.conf for time extraction.

TIME_FORMAT = %T.%3N
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 20

Please read - https://docs.splunk.com/Documentation/Splunk/8.2.6/Data/HowSplunkextractstimestamps 

I hope this resolves your problem. If so please consider upvoting!!!

[PickleRick]

If you have the date in source field, you can as well use ingest-time eval and metadata manipulation to modify the _time field explicitly.

[VatsalJagani]

@PickleRick - 😊That is new to me that you can change timestamp as well with metadata manipulation.
Can you please share any document or config example for that? I would really appreciate that.

[PickleRick · Accepted solution]

OK, that was a bit of a long shot since I've never done it myself but.

1) The time parsing comes as one of the first stages of event processing during ingestion

2) You can manipulate metadata with transforms

3) You can use ingest-time evals to modify values not only with simple regexes but also with quite a significant range of normal splunk operations.

See the https://conf.splunk.com/files/2020/slides/PLA1154C.pdf for example.

[PickleRick]

If you want to split the file into separate event per each line and use the date from the file's header... then you can't do it by splunk alone. You can't "parse and then split" events. You can't "carry" some information from one event over to another one. Each event is parsed and interpreted separately. You'd need to have some script which would prepare your file by cutting and pasting the date into each line and only ingest such prepared file, not the original one.