How to extract a particular string from the event logs

[kumarnis45]

Hi Guys,

    I have a scenario where i need to extract the file name from the event logs. The Event log first line looks like below.

Event Log:

[INFO] 2021-09-30T00:04:17.052Z 8d5eb00a-d033-49a9-9d0f-c61011e4ae51 {"Records": [{"eventVersion": }]

Now i need to write a rex query to extract the file name "8d5eb00a-d033-49a9-9d0f-c61011e4ae51" from above event log. This file name changes for the every search query along with the timestamp.

Can someone suggest me how to resolve this?

Thanks.

[richgalloway]

See if this helps.  It looks for the milliseconds in the timestamp, a time zone indicator, and some white space then takes everything up to the next white space to be the file name.

... | rex "\.\d{3}\w+\s(?<filename>\S+)"

[kumarnis45]

@richgalloway 

I have tried it as you suggested. But its returning the same result as before.  I have pasted the logs below. I have to remove the timestamp and return the just 12345 as the output.

Logs are:

[INFO] 2021-10-02T00:09:50.398Z 12345 {"Records": [{"eventVersion": "2.1", "eventSource": "aws:s3", "awsRegion": "us-east-1", "eventTime": "2021-10-02T00:09:42.743Z", "eventName": "ObjectCreated:Put", "userIdentity":

[richgalloway]

My query works in regex101.com with both of your sample events.  If it's not working for you then we need more information.  Please share the full query you are using to extract the string.  What is the result (the OP didn't say)?

[kumarnis45]

the full query is looks like below:

I am looking for source lambda logs with a json file to return a reqid. 

source=/aws/lambda/sample test.json | rex "\.\d{3}\w+\s(?<file>\S+)"

In log events it returns a reqid '12345' as below. There is some space after timestamp and after reqid.

Logs are:

[INFO] 2021-10-02T00:09:50.398Z 12345 {"Records": [{"eventVersion": "2.1", "eventSource": "aws:s3", "awsRegion": "us-east-1", "eventTime": "2021-10-02T00:09:42.743Z", "eventName": "ObjectCreated:Put", "userIdentity":

I hope this is clear now. let me know if i need to add extra details

[richgalloway]

Based on the latest response and the OP, it appears the two regular expressions provided do what was asked.

Please explain how "12345" is not what is needed then tell us what is needed from each event.

[kumarnis45]

please check the attached picture. It has the command you suggested me to run with the OP

[kumarnis45]

@richgalloway 

source=/aws/lambda/sample test.json 

source=/aws/lambda/sample test.json | rex "\.\d{3}\w+\s(?<file>\S+)"

Both the above commands returning the same result as below.

Logs are:

[INFO] 2021-10-02T00:09:50.398Z 12345 {"Records": [{"eventVersion": "2.1", "eventSource": "aws:s3", "awsRegion": "us-east-1", "eventTime": "2021-10-02T00:09:42.743Z", "eventName": "ObjectCreated:Put", "userIdentity":

Adding the rex to extract and display the '12345' not working in my scenario. do wee need to update rex command?

Thanks.

[richgalloway]

As @ITWhisperer said, there appears to be more than one space between the timestamp and the regid.  Try this command.

... | rex "\.\d{3}\w+\s+(?<file>\S+)"

[ITWhisperer]

| rex "\[INFO\]\s\S+\s(?<filename>\S+)\s"

[kumarnis45]

I have tried it as you suggested. But its returning the same result as before.  I have pasted the logs below. I have to remove the timestamp and return the just 12345 as the output.

Logs are:

[INFO] 2021-10-02T00:09:50.398Z 12345 {"Records": [{"eventVersion": "2.1", "eventSource": "aws:s3", "awsRegion": "us-east-1", "eventTime": "2021-10-02T00:09:42.743Z", "eventName": "ObjectCreated:Put", "userIdentity":

[kumarnis45]

@ITWhisperer ,

   I still couldn't figure it out to get this work. Tried using rex with different possibilities but none of them are working. can you please suggest a better solution?

Thanks. 

[ITWhisperer]

What exactly do you get when you use the command suggested?

Don't forget, the only information we have is what you post! The more information you can give us, the easier it will be for us to help.

[kumarnis45]

Please check the attached screenshot

[ITWhisperer · Accepted solution]

It looks like you have multiple white spaces - try this

| rex "\[INFO\]\s+\S+\s+(?<filename>\S+)\s"

[kumarnis45]

This is working. adding | table filename after rex fixed it.

| rex "\[INFO\]\s+\S+\s+(?<filename>\S+)\s" | table filename

[kumarnis45]

Not working though. Returning the same result