topic Re: How to fetch the keywords from raw logs in Dashboards & Visualizations

How to fetch the keywords from raw logs?

aditsss — Mon, 17 Jul 2023 18:44:21 GMT

Hi Team,

How we can fetch the below keywords from raw logs:

2023-06-29 09:41:53.884 [INFO ] [pool-2-thread-1] ArchivalProcessor - finished reading file /absin/TRIM.ARCH.D062923.T052525

2023-07-13 02:42:02.915 [INFO ] [pool-2-thread-1] FileSensor - Start Reading Account balance Data File, QACDU.D062623.T065000

2023-07-13 18:53:10.226 [INFO ] [pool-5-thread-1] FileSensor - Completed Account balance file processing, QACDU.D062623.T065000 records processed: 105932244, Kafka counter: 0

Re: How to fetch the keywords from raw logs

gcusello — Fri, 14 Jul 2023 14:01:02 GMT

Hi @aditsss,

sorry but it isn't clear the rule of your extraction: do you want the string after the minus sign?

If this is your requirement, please try:

| rex "-\s+(?<message>.*)"

that you can test at https://regex101.com/r/JUWcZh/1

Ciao.

Giuseppe

Re: How to fetch the keywords from raw logs

aditsss — Fri, 14 Jul 2023 14:08:18 GMT

@gcusello

For this particular logger I just need file name:

2023-06-29 09:41:53.884 [INFO ] [pool-2-thread-1] ArchivalProcessor - finished reading file /absin/TRIM.ARCH.D062923.T052525

How Can I fetch it.

Re: How to fetch the keywords from raw logs

gcusello — Fri, 14 Jul 2023 14:14:32 GMT

Hi @aditsss,

please try this:

| rex "-\s+(?<message>.*)" | rex "\/(?<message1>.+)$ | eval message=if(match(message,"finished reading file%",message1,message)

Ciao.

Giuseppe

Re: How to fetch the keywords from raw logs

aditsss — Fri, 14 Jul 2023 14:22:44 GMT

@gcusello

I tried this:

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 source!="/var/log/messages" "finished reading file"
| rex "-\s+(?<message>.*)" | rex "\/(?<message1>.+)$"|eval message=if(match(message,"finished reading file%",message1,message))|stats count by message1

getting this result:

Error in 'EvalCommand': The arguments to the 'match' function are invalid.

Re: How to fetch the keywords from raw logs

gcusello — Fri, 14 Jul 2023 14:40:24 GMT

Hi @aditsss,

change parenthesis and use message not message1:

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 source!="/var/log/messages" "finished reading file" | rex "-\s+(?<message>.*)" | rex "\/(?<message1>.+)$" | eval message=if(match(message,"finished reading file%"),message1,message) | stats count by message

Ciao.

Giuseppe

Re: How to fetch the keywords from raw logs

aditsss — Fri, 14 Jul 2023 15:19:47 GMT

@gcusello

I just want file name

2023-06-29 09:41:53.884 [INFO ] [pool-2-thread-1] ArchivalProcessor - finished reading file /absin/TRIM.ARCH.D062923.T052525

with this query :

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 source!="/var/log/messages" "finished reading file"
| rex "-\s+(?<message>.*)"
| rex "\/(?<message1>.+)$"
| eval message=if(match(message,"finished reading file%"),message1,message)
| stats count by message1

I am getting result like this:

absin/TRIM.ARCH.D062223.T081112

Re: How to fetch the keywords from raw logs

gcusello — Sat, 15 Jul 2023 09:06:24 GMT

Hi @aditsss,

as I said, use message, not message1 in the stats command, anyeay, please try this:

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 source!="/var/log/messages" "finished reading file" | rex "-\s+(?<message>.*)" | rex "\/\w+\/(?<message1>.*)$" | eval message=if(match(message,"finished reading file%"),message1,message) | stats count by message

Ciao.

Giuseppe