topic Re: Scraping Start Times in Knowledge Management

Scraping Start Times

Regleston — Wed, 20 Feb 2019 19:22:18 GMT

I am trying to extract the time taken for a process to execute from my logs. This is they syntax of the log:

Time taken: 1.422 seconds

I have been through the message boards on here and tried dozens of different suggestions from similar issues but none of the rex/regex combinations have worked and just given me the time taken. Can someone please help me out?

Re: Scraping Start Times

Vijeta — Wed, 20 Feb 2019 20:34:19 GMT

Do you want to extract the value of time (1.422) from the logs?

Re: Scraping Start Times

Regleston — Wed, 20 Feb 2019 20:45:42 GMT

Yes, I'm just trying to get what every time shows in that area.

Re: Scraping Start Times

pkeenan87 — Wed, 20 Feb 2019 20:49:59 GMT

Try this regex:

Time\staken:\s(?[^\s]+)

Re: Scraping Start Times

adonio — Wed, 20 Feb 2019 21:12:50 GMT

try this anywhere:

| makeresults count=1
| eval data = "Time taken: 1.422 seconds;;;Time taken: 1.712 seconds;;;Time taken: 1.333 seconds;;;Time taken: 1.290 seconds"
| makemv delim=";;;" data
| mvexpand data
| rename COMMENT as "the above generates data below is the solution" 
| rex field=data "taken:\s+(?<time_taken>[^\s]+)"

hope it helps

Re: Scraping Start Times

Regleston — Wed, 20 Feb 2019 21:47:17 GMT

It seems to work great with your sample data but for some reason when I add it to the end of my base search I get the same result as I would if I didn't add it.

Re: Scraping Start Times

adonio — Wed, 20 Feb 2019 22:21:32 GMT

share your base search and a sample full event

Re: Scraping Start Times

Regleston — Thu, 21 Feb 2019 14:03:02 GMT

index=app_hdfs host=FQHN source+/opt/hdp/dna-sqoop-ingest/log/mls-ingest-logs/ingest-mls-summary*.log | rex field=data "taken:\s+(?[^\s]+)"

Time taken: 1.422 seconds
host = FQHN source = /opt/hdp/dna-sqoop-ingest/log/mls-ingest-logs/ingest-mls-summary.log sourcetype = ingest-mls-summary

Re: Scraping Start Times

Regleston — Thu, 21 Feb 2019 16:51:23 GMT

(Previous messages got removed for some reason)
That works great with your sample data but with my base search does nothing.

My search query:
index= host= source=//mls-ingest-logs/ingest-mls-summary*.log| rex field=data "taken:\s+(?[^\s]+)"

Log Event:
Time taken: 1.422 seconds
host = source = //ingest-mls-summary.log sourcetype = ingest-mls-summary

Re: Scraping Start Times

adonio — Thu, 21 Feb 2019 17:54:51 GMT

the field data is in my example, you can remove it for your query
.... | rex field=_raw "taken:\s+(?<time_taken>[^\s]+)"

Re: Scraping Start Times

Regleston — Thu, 21 Feb 2019 17:58:57 GMT

Oh the first thing after the search made no difference was to switch it to "_raw", just reverted it back for the comment.

Re: Scraping Start Times

adonio — Thu, 21 Feb 2019 18:12:54 GMT

ok, the query i provided first is a dummy query that creates fake data and shows that the regex works

try it top make sure its fine.
on your real data, use this:
index=YOUR_INDEX_NAME_HERE host=YOUR_HOST_NAME_HERE source=//mls-ingest-logs/ingest-mls-summary*.log| rex field=_raw "taken:\s+(?[^\s]+)"