Getting Data In

How to extract Date and time logged to different locations in the same file?

NickyCarlisle
Engager

Hello, I have a log file where the date is at the top of the log and the time for each event is at the start of each line, so something like this:

-- Log Continued 03/28/2022 00:00:00.471 --
00:00:36.526 xxxxx
00:04:01.809 xxxxx
00:04:09.267 xxxxx
00:10:19.039 xxxxx

How would I extract the date/ time using props.conf or similar?

0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

OK, that was a bit of a long shot since I've never done it myself but.

1) The time parsing comes as one of the first stages of event processing during ingestion

2) You can manipulate metadata with transforms

3) You can use ingest-time evals to modify values not only with simple regexes but also with quite a significant range of normal splunk operations.

See the https://conf.splunk.com/files/2020/slides/PLA1154C.pdf for example.

View solution in original post

VatsalJagani
SplunkTrust
SplunkTrust

You cannot extract the date from the header of the file but I think if you date in the file name then you should be able to extract the date from the file name and time from the event itself. This should work as far as you have the proper date format as part of your file/source name.

VatsalJagani_0-1649226282336.png

Use below for props.conf for time extraction.

 

TIME_FORMAT = %T.%3N
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 20

 

Please read - https://docs.splunk.com/Documentation/Splunk/8.2.6/Data/HowSplunkextractstimestamps 

 

I hope this resolves your problem. If so please consider upvoting!!!

PickleRick
SplunkTrust
SplunkTrust

If you have the date in source field, you can as well use ingest-time eval and metadata manipulation to modify the _time field explicitly.

VatsalJagani
SplunkTrust
SplunkTrust

@PickleRick - 😊That is new to me that you can change timestamp as well with metadata manipulation.
Can you please share any document or config example for that? I would really appreciate that.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK, that was a bit of a long shot since I've never done it myself but.

1) The time parsing comes as one of the first stages of event processing during ingestion

2) You can manipulate metadata with transforms

3) You can use ingest-time evals to modify values not only with simple regexes but also with quite a significant range of normal splunk operations.

See the https://conf.splunk.com/files/2020/slides/PLA1154C.pdf for example.

PickleRick
SplunkTrust
SplunkTrust

If you want to split the file into separate event per each line and use the date from the file's header... then you can't do it by splunk alone. You can't "parse and then split" events. You can't "carry" some information from one event over to another one. Each event is parsed and interpreted separately. You'd need to have some script which would prepare your file by cutting and pasting the date into each line and only ingest such prepared file, not the original one.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

🍂 Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...