Knowledge Management

Scraping Start Times

Regleston
New Member

I am trying to extract the time taken for a process to execute from my logs. This is they syntax of the log:

Time taken: 1.422 seconds

I have been through the message boards on here and tried dozens of different suggestions from similar issues but none of the rex/regex combinations have worked and just given me the time taken. Can someone please help me out?

Tags (1)
0 Karma

adonio
Ultra Champion

try this anywhere:

| makeresults count=1
| eval data = "Time taken: 1.422 seconds;;;Time taken: 1.712 seconds;;;Time taken: 1.333 seconds;;;Time taken: 1.290 seconds"
| makemv delim=";;;" data
| mvexpand data
| rename COMMENT as "the above generates data below is the solution" 
| rex field=data "taken:\s+(?<time_taken>[^\s]+)"

hope it helps

0 Karma

Regleston
New Member

(Previous messages got removed for some reason)
That works great with your sample data but with my base search does nothing.

My search query:
index= host= source=//mls-ingest-logs/ingest-mls-summary*.log| rex field=data "taken:\s+(?[^\s]+)"

Log Event:
Time taken: 1.422 seconds
host = source = //ingest-mls-summary.log sourcetype = ingest-mls-summary

0 Karma

adonio
Ultra Champion

the field data is in my example, you can remove it for your query
.... | rex field=_raw "taken:\s+(?<time_taken>[^\s]+)"

0 Karma

Regleston
New Member

Oh the first thing after the search made no difference was to switch it to "_raw", just reverted it back for the comment.

0 Karma

adonio
Ultra Champion

ok, the query i provided first is a dummy query that creates fake data and shows that the regex works

try it top make sure its fine.
on your real data, use this:
index=YOUR_INDEX_NAME_HERE host=YOUR_HOST_NAME_HERE source=//mls-ingest-logs/ingest-mls-summary*.log| rex field=_raw "taken:\s+(?[^\s]+)"

0 Karma

Regleston
New Member

It seems to work great with your sample data but for some reason when I add it to the end of my base search I get the same result as I would if I didn't add it.

0 Karma

adonio
Ultra Champion

share your base search and a sample full event

0 Karma

Regleston
New Member

index=app_hdfs host=FQHN source+/opt/hdp/dna-sqoop-ingest/log/mls-ingest-logs/ingest-mls-summary*.log | rex field=data "taken:\s+(?[^\s]+)"

Time taken: 1.422 seconds
host = FQHN source = /opt/hdp/dna-sqoop-ingest/log/mls-ingest-logs/ingest-mls-summary.log sourcetype = ingest-mls-summary

0 Karma

pkeenan87
Communicator

Try this regex:

Time\staken:\s(?[^\s]+)

0 Karma

Vijeta
Influencer

Do you want to extract the value of time (1.422) from the logs?

0 Karma

Regleston
New Member

Yes, I'm just trying to get what every time shows in that area.

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...