How to fetch the same variable with different value from the logs?

[aditsss]

Hi Team,

I am getting below raw logs:

2023-07-29 10:39:52.949 [INFO ] [Thread-3] AssociationProcessor - compareTransformStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=19020051, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=07/28/2023, fileName=SETTLEMENT_TRANSFORM_MERGE, totalAchCurrOutstBalAmt=0.0, totalAchBalLastStmtAmt=0.0, totalClosingBal=7.100761644428E10, sourceName=null, version=1, associationStats={}] ---- controlFileData: ControlFileData [fileName=SETTLEMENT_TRANSFORM_ASSOCIATION, busDate=07/28/2023, fileSequenceNum=0, totalBalanceLastStmt=0.0, totalCurrentOutstBal=0.0, totalRecordsWritten=19020051, totalRecords=0, totalClosingBal=7.100761644428E10]

I want to fetch the highlighted information the query I am trying is below:

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 sourcetype = "600000304_gg_abs_ipc2" " AssociationProcessor* associationStats={}] ---- controlFileData:ControlFileData " source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" |rex " AssociationProcessor* associationStats={}] ---- controlFileData:ControlFileData busDt=(?<busDt>),fileName=(?<fileName>),totalClosingBal=(?<totalClosingBal>)"|table _time  busDt fileName totalClosingBal|sort _time

But I am getting this file name in my statistics "fileName=SETTLEMENT_TRANSFORM_MERGE"  rather I want the one inside Association Stats "SETTLEMENT_TRANSFORM_ASSOCIATION"

Can someone gu

[ITWhisperer]

Try testing you regex in regex101.com to see what it is doing and hopefully figure out what needs to change.

I have made a start for you https://regex101.com/r/Uylo38/1

Hint: * means zero or more of the previous character (or match group) and [ is a special character in regex so would need to be escaped if you want to match with an actual [ in your string.

[aditsss]

@ITWhisperer 

Can you help me here  I need to sow this panel tomorrow

[ITWhisperer]

Your regex statement is doing this:

AssociationProcesso

matches the characters AssociationProcesso literally (case sensitive)

r

matches the character r with index 11410 (7216 or 1628) literally (case sensitive)

* matches the previous token between zero and unlimited times, as many times as possible, giving back as needed (greedy)

associationStats={}]----controlFileData:ControlFileDatabusDt=

matches the characters associationStats={}] ---- controlFileData:ControlFileData busDt= literally (case sensitive)

Named Capture Group busDt

(?<busDt>)

 — always finds a zero-length match

,fileName=

matches the characters ,fileName= literally (case sensitive)

Named Capture Group fileName

(?<fileName>)

 — always finds a zero-length match

,totalClosingBal=

matches the characters ,totalClosingBal= literally (case sensitive)

Named Capture Group totalClosingBal

(?<totalClosingBal>)

 — always finds a zero-length match

[aditsss]

@ITWhisperer 

what regex I should use please guide 

I tried with this:

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 sourcetype = "600000304_gg_abs_ipc2" " associationStats={}] ---- controlFileData: ControlFileData " source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" |rex " associationStats={}] ---- controlFileData: ControlFileData " busDt=(?<busDt>),fileName=(?<fileName>),totalClosingBal=(?<totalClosingBal>)"|table _time  busDt fileName totalClosingBal|sort _time

But its taking the file other log also that is why I use AssociationProcessor*

please guide

Below is the screenshot I want to fetch first one

[ITWhisperer]

You can still use AssociationProcessor in your search filter, it doesn't have to also be in your regex

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 sourcetype = "600000304_gg_abs_ipc2" " AssociationProcessor* associationStats={}] ---- controlFileData:ControlFileData " source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" 
|rex " busDt=(?<busDt>),fileName=(?<fileName>),totalClosingBal=(?<totalClosingBal>)"
|table _time  busDt fileName totalClosingBal
|sort _time

Now you just need to fix the regex - for example, do the strings actually match up with your events? what characters are you tying to capture in the capture groups?

[aditsss]

@ITWhisperer 

I want filename BusDate and closing balance

[ITWhisperer]

What pattern would find those characters in the capture groups?

Try doing just the first one until you get that right, then move on to the next one - try this out in regex101.com as it tells you what your pattern is matching against.

[aditsss]

@ITWhisperer 

I tried below query:

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 "AssociationProcessor* associationStats={}] ---- controlFileData: ControlFileData" source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log"|rex " busDate=(?<busDate>),fileName=(?<fileName>),totalClosingBal=(?<totalClosingBal>)"
|table _time busDate fileName totalClosingBal
|sort _time

Getting below result:

This is not the correct result 

Below is the raw log @ITWhisperer its capturing file name as "fileName=SETTLEMENT_TRANSFORM_MERGE" I WANT FILE NAME TO BE THE ONE PRESENT INSIDE 

associationStats={} THAT IS "fileName=SETTLEMENT_TRANSFORM_ASSOCIATION"

2023-07-29 10:39:52.949 [INFO ] [Thread-3] AssociationProcessor - compareTransformStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=19020051, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=07/28/2023, fileName=SETTLEMENT_TRANSFORM_MERGE, totalAchCurrOutstBalAmt=0.0, totalAchBalLastStmtAmt=0.0, totalClosingBal=7.100761644428E10, sourceName=null, version=1, associationStats={}] ---- controlFileData: ControlFileData [fileName=SETTLEMENT_TRANSFORM_ASSOCIATION, busDate=07/28/2023, fileSequenceNum=0, totalBalanceLastStmt=0.0, totalCurrentOutstBal=0.0, totalRecordsWritten=19020051, totalRecords=0, totalClosingBal=7.100761644428E10]

[ITWhisperer]

Your rex is not capturing anything, you have not pattern inside you capture groups for rex to extract against. The value you are seeing for these fields is the value from the index search. You need to modify the rex so that it finds the right place in the log to start the extract from (this is called an anchor), then define what pattern you want to extract into the capture group / field. Look at what regex101.com is telling you is happening for your regex.

[aditsss]

@ITWhisperer 

I am not sure exactly what rex need to be used here could you please guide

[ITWhisperer]

OK assuming you anchor to the right "fileName=", how would you describe the characters you want to be included in the fileName field?

[aditsss]

@ITWhisperer 

Below file name I want:

fileName=SETTLEMENT_TRANSFORM_ASSOCIATION

AssociationProcessor - compareTransformStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=19020051, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=07/28/2023, fileName=SETTLEMENT_TRANSFORM_MERGE, totalAchCurrOutstBalAmt=0.0, totalAchBalLastStmtAmt=0.0, totalClosingBal=7.100761644428E10, sourceName=null, version=1, associationStats={}] ---- controlFileData: ControlFileData [fileName=SETTLEMENT_TRANSFORM_ASSOCIATION, busDate=07/28/2023, fileSequenceNum=0, totalBalanceLastStmt=0.0, totalCurrentOutstBal=0.0, totalRecordsWritten=19020051, totalRecords=0, totalClosingBal=7.100761644428E10]

I want to display the information inside associationStats={}] ---- controlFileData: ControlFileData [

[ITWhisperer · Accepted solution]

|rex "fileName=(?<fileName>SETTLEMENT_TRANSFORM_ASSOCIATION)"