topic Re: How to fetch the same variable with different value from the logs in Dashboards & Visualizations

How to fetch the same variable with different value from the logs?

aditsss — Mon, 31 Jul 2023 18:44:34 GMT

Hi Team,

I am getting below raw logs:

2023-07-29 10:39:52.949 [INFO ] [Thread-3] AssociationProcessor - compareTransformStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=19020051, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=07/28/2023, fileName=SETTLEMENT_TRANSFORM_MERGE, totalAchCurrOutstBalAmt=0.0, totalAchBalLastStmtAmt=0.0, totalClosingBal=7.100761644428E10, sourceName=null, version=1, associationStats={}] ---- controlFileData: ControlFileData [fileName=SETTLEMENT_TRANSFORM_ASSOCIATION, busDate=07/28/2023, fileSequenceNum=0, totalBalanceLastStmt=0.0, totalCurrentOutstBal=0.0, totalRecordsWritten=19020051, totalRecords=0, totalClosingBal=7.100761644428E10]

I want to fetch the highlighted information the query I am trying is below:

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 sourcetype = "600000304_gg_abs_ipc2" " AssociationProcessor* associationStats={}] ---- controlFileData:ControlFileData " source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" |rex " AssociationProcessor* associationStats={}] ---- controlFileData:ControlFileData busDt=(?<busDt>),fileName=(?<fileName>),totalClosingBal=(?<totalClosingBal>)"|table _time busDt fileName totalClosingBal|sort _time

But I am getting this file name in my statistics "fileName=SETTLEMENT_TRANSFORM_MERGE" rather I want the one inside Association Stats "SETTLEMENT_TRANSFORM_ASSOCIATION"

Can someone gu

Re: How to fetch the same variable with different value from the logs

ITWhisperer — Sun, 30 Jul 2023 09:25:15 GMT

Try testing you regex in regex101.com to see what it is doing and hopefully figure out what needs to change.

I have made a start for you https://regex101.com/r/Uylo38/1

Hint: * means zero or more of the previous character (or match group) and [ is a special character in regex so would need to be escaped if you want to match with an actual [ in your string.

Re: How to fetch the same variable with different value from the logs

aditsss — Sun, 30 Jul 2023 10:00:50 GMT

@ITWhisperer

Can you help me here I need to sow this panel tomorrow

Re: How to fetch the same variable with different value from the logs

ITWhisperer — Sun, 30 Jul 2023 10:13:48 GMT

Your regex statement is doing this:

AssociationProcesso

matches the characters AssociationProcesso literally (case sensitive)

matches the character r with index 11410 (7216 or 1628) literally (case sensitive)

* matches the previous token between zero and unlimited times, as many times as possible, giving back as needed (greedy)

associationStats={}]----controlFileData:ControlFileDatabusDt=

matches the characters associationStats={}] ---- controlFileData:ControlFileData busDt= literally (case sensitive)

Named Capture Group busDt

(?<busDt>)

— always finds a zero-length match

,fileName=

matches the characters ,fileName= literally (case sensitive)

Named Capture Group fileName

(?<fileName>)

— always finds a zero-length match

,totalClosingBal=

matches the characters ,totalClosingBal= literally (case sensitive)

Named Capture Group totalClosingBal

(?<totalClosingBal>)

— always finds a zero-length match

Re: How to fetch the same variable with different value from the logs

aditsss — Sun, 30 Jul 2023 10:32:40 GMT

@ITWhisperer

what regex I should use please guide

I tried with this:

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 sourcetype = "600000304_gg_abs_ipc2" " associationStats={}] ---- controlFileData: ControlFileData " source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" |rex " associationStats={}] ---- controlFileData: ControlFileData " busDt=(?<busDt>),fileName=(?<fileName>),totalClosingBal=(?<totalClosingBal>)"|table _time busDt fileName totalClosingBal|sort _time

But its taking the file other log also that is why I use AssociationProcessor*

please guide

Below is the screenshot I want to fetch first one

Re: How to fetch the same variable with different value from the logs

ITWhisperer — Sun, 30 Jul 2023 10:33:10 GMT

You can still use AssociationProcessor in your search filter, it doesn't have to also be in your regex

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 sourcetype = "600000304_gg_abs_ipc2" " AssociationProcessor* associationStats={}] ---- controlFileData:ControlFileData " source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" |rex " busDt=(?<busDt>),fileName=(?<fileName>),totalClosingBal=(?<totalClosingBal>)" |table _time busDt fileName totalClosingBal |sort _time

Now you just need to fix the regex - for example, do the strings actually match up with your events? what characters are you tying to capture in the capture groups?

Re: How to fetch the same variable with different value from the logs

aditsss — Sun, 30 Jul 2023 10:36:55 GMT

@ITWhisperer

I want filename BusDate and closing balance

Re: How to fetch the same variable with different value from the logs

ITWhisperer — Sun, 30 Jul 2023 10:39:35 GMT

What pattern would find those characters in the capture groups?

Try doing just the first one until you get that right, then move on to the next one - try this out in regex101.com as it tells you what your pattern is matching against.

Re: How to fetch the same variable with different value from the logs

aditsss — Sun, 30 Jul 2023 10:45:03 GMT

@ITWhisperer

I tried below query:

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 "AssociationProcessor* associationStats={}] ---- controlFileData: ControlFileData" source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log"|rex " busDate=(?<busDate>),fileName=(?<fileName>),totalClosingBal=(?<totalClosingBal>)"
|table _time busDate fileName totalClosingBal
|sort _time

Getting below result:

This is not the correct result

Below is the raw log @ITWhisperer its capturing file name as "fileName=SETTLEMENT_TRANSFORM_MERGE" I WANT FILE NAME TO BE THE ONE PRESENT INSIDE

associationStats={} THAT IS "fileName=SETTLEMENT_TRANSFORM_ASSOCIATION"

2023-07-29 10:39:52.949 [INFO ] [Thread-3] AssociationProcessor - compareTransformStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=19020051, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=07/28/2023, fileName=SETTLEMENT_TRANSFORM_MERGE, totalAchCurrOutstBalAmt=0.0, totalAchBalLastStmtAmt=0.0, totalClosingBal=7.100761644428E10, sourceName=null, version=1, associationStats={}] ---- controlFileData: ControlFileData [fileName=SETTLEMENT_TRANSFORM_ASSOCIATION, busDate=07/28/2023, fileSequenceNum=0, totalBalanceLastStmt=0.0, totalCurrentOutstBal=0.0, totalRecordsWritten=19020051, totalRecords=0, totalClosingBal=7.100761644428E10]

Re: How to fetch the same variable with different value from the logs

ITWhisperer — Sun, 30 Jul 2023 10:56:50 GMT

Your rex is not capturing anything, you have not pattern inside you capture groups for rex to extract against. The value you are seeing for these fields is the value from the index search. You need to modify the rex so that it finds the right place in the log to start the extract from (this is called an anchor), then define what pattern you want to extract into the capture group / field. Look at what regex101.com is telling you is happening for your regex.

Re: How to fetch the same variable with different value from the logs

aditsss — Sun, 30 Jul 2023 11:27:57 GMT

@ITWhisperer

I am not sure exactly what rex need to be used here could you please guide

Re: How to fetch the same variable with different value from the logs

ITWhisperer — Sun, 30 Jul 2023 12:05:48 GMT

OK assuming you anchor to the right "fileName=", how would you describe the characters you want to be included in the fileName field?

Re: How to fetch the same variable with different value from the logs

aditsss — Sun, 30 Jul 2023 15:18:00 GMT

@ITWhisperer

Below file name I want:

fileName=SETTLEMENT_TRANSFORM_ASSOCIATION

AssociationProcessor - compareTransformStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=19020051, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=07/28/2023, fileName=SETTLEMENT_TRANSFORM_MERGE, totalAchCurrOutstBalAmt=0.0, totalAchBalLastStmtAmt=0.0, totalClosingBal=7.100761644428E10, sourceName=null, version=1, associationStats={}] ---- controlFileData: ControlFileData [fileName=SETTLEMENT_TRANSFORM_ASSOCIATION, busDate=07/28/2023, fileSequenceNum=0, totalBalanceLastStmt=0.0, totalCurrentOutstBal=0.0, totalRecordsWritten=19020051, totalRecords=0, totalClosingBal=7.100761644428E10]

I want to display the information inside associationStats={}] ---- controlFileData: ControlFileData [

Re: How to fetch the same variable with different value from the logs

ITWhisperer — Sun, 30 Jul 2023 16:56:25 GMT

|rex "fileName=(?<fileName>SETTLEMENT_TRANSFORM_ASSOCIATION)"