topic Re: How to exclude other words from one log in Dashboards & Visualizations

How to exclude other words from one log

aditsss — Thu, 31 Aug 2023 17:52:36 GMT

Hi Team,

I have below row logs:

2023-08-30 07:43:29.000 [INFO ] [Thread-18] StatisticBalancer - statisticData: StatisticData [selectedDataSet=13283520, rejectedDataSet=0, totalOutputRecords=20670402, totalInputRecords=0, fileSequenceNum=9226, fileHeaderBusDt=08/29/2023, busDt=08/29/2023, fileName=TRIM.UNB.D082923.T045920]

2023-08-30 05:36:30.678 [INFO ] [Thread-19] StatisticBalancer - statisticData: StatisticData [selectedDataSet=27, rejectedDataSet=0, totalOutputRecords=27, totalInputRecords=0, fileSequenceNum=6395, fileHeaderBusDt=08/29/2023, busDt=08/29/2023, fileName=TRIM.CNX.D082923.T052656]

I want to fetch records only for highlighted file not for other files but I am getting for both the files.

My current query:

index="600000304_d_gridgain_idx*" sourcetype =$Regions$ source="/amex/app/gfp-settlement-raw/logs/gfp-settlement-raw.log" "StatisticBalancer - statisticData: StatisticData" |rex "totalOutputRecords=(?<totalOutputRecords>),busDt=(?<busDt>),fileName=(?<fileName>),totalAchCurrOutstBalAmt=(?<totalAchCurrOutstBalAmt>),totalAchBalLastStmtAmt=(?<totalAchBalLastStmtAmt>),totalClosingBal=(?<totalClosingBal>),totalRecordsWritten=(?<totalRecordsWritten>),totalRecords=(?<totalRecords>)" |table busDt fileName totalAchCurrOutstBalAmt totalAchBalLastStmtAmt totalClosingBal totalRecordsWritten totalRecords

Re: How to exclude other words from one log

richgalloway — Thu, 31 Aug 2023 17:58:28 GMT

There are two ways to exclude events containing certain words. The first is to put the word(s) in the base search ( the part before the first |) preceded by "NOT". It may be easier to put desired words here if that number is smaller. This won't work if the work must be in a specific field this not yet extracted.

index="600000304_d_gridgain_idx*" sourcetype =$Regions$ source="/amex/app/gfp-settlement-raw/logs/gfp-settlement-raw.log" "StatisticBalancer - statisticData: StatisticData" "TRIM.UNB.D082923.T045920" |rex "totalOutputRecords=(?<totalOutputRecords>),busDt=(?<busDt>),fileName=(?<fileName>),totalAchCurrOutstBalAmt=(?<totalAchCurrOutstBalAmt>),totalAchBalLastStmtAmt=(?<totalAchBalLastStmtAmt>),totalClosingBal=(?<totalClosingBal>),totalRecordsWritten=(?<totalRecordsWritten>),totalRecords=(?<totalRecords>)" |table busDt fileName totalAchCurrOutstBalAmt totalAchBalLastStmtAmt totalClosingBal totalRecordsWritten totalRecords

The other way is to use the search or where command to filter out events with the offending words (or keep those with desired words).

index="600000304_d_gridgain_idx*" sourcetype =$Regions$ source="/amex/app/gfp-settlement-raw/logs/gfp-settlement-raw.log" "StatisticBalancer - statisticData: StatisticData" |rex "totalOutputRecords=(?<totalOutputRecords>),busDt=(?<busDt>),fileName=(?<fileName>),totalAchCurrOutstBalAmt=(?<totalAchCurrOutstBalAmt>),totalAchBalLastStmtAmt=(?<totalAchBalLastStmtAmt>),totalClosingBal=(?<totalClosingBal>),totalRecordsWritten=(?<totalRecordsWritten>),totalRecords=(?<totalRecords>)" | where fileName="TRIM.UNB.D082923.T045920" |table busDt fileName totalAchCurrOutstBalAmt totalAchBalLastStmtAmt totalClosingBal totalRecordsWritten totalRecords

Re: How to exclude other words from one log

aditsss — Wed, 06 Sep 2023 11:53:09 GMT

Hi @richgalloway

I tried with this query but not able to see any result:

index="abc" sourcetype =$Regions$ source="/amex/app/gfp-settlement-raw/logs/gfp-settlement-raw.log" "StatisticBalancer - statisticData: StatisticData"
|rex "totalOutputRecords=(?<totalOutputRecords>),busDt=(?<busDt>),fileName=(?<fileName>),totalAchCurrOutstBalAmt=(?<totalAchCurrOutstBalAmt>),totalAchBalLastStmtAmt=(?<totalAchBalLastStmtAmt>),totalClosingBal=(?<totalClosingBal>),totalRecordsWritten=(?<totalRecordsWritten>),totalRecords=(?<totalRecords>)"
| where fileName="TRIM.UNB.D082923.T045920"
|table busDt fileName totalAchCurrOutstBalAmt totalAchBalLastStmtAmt totalClosingBal totalRecordsWritten totalRecords

Re: How to exclude other words from one log

richgalloway — Wed, 06 Sep 2023 12:55:37 GMT

It's possible no events have the expected fileName value during the selected time range. Try removing the where command to see if results are shown. If they are, then examine the events closely to ensure they are filtered as desired.