Dashboards & Visualizations

How to fetch the same variable with different value from the logs?

aditsss
Motivator

Hi Team,

I am getting below raw logs:

2023-07-29 10:39:52.949 [INFO ] [Thread-3] AssociationProcessor - compareTransformStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=19020051, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=07/28/2023, fileName=SETTLEMENT_TRANSFORM_MERGE, totalAchCurrOutstBalAmt=0.0, totalAchBalLastStmtAmt=0.0, totalClosingBal=7.100761644428E10, sourceName=null, version=1, associationStats={}] ---- controlFileData: ControlFileData [fileName=SETTLEMENT_TRANSFORM_ASSOCIATION, busDate=07/28/2023, fileSequenceNum=0, totalBalanceLastStmt=0.0, totalCurrentOutstBal=0.0, totalRecordsWritten=19020051, totalRecords=0, totalClosingBal=7.100761644428E10]

I want to fetch the highlighted information the query I am trying is below:

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 sourcetype = "600000304_gg_abs_ipc2" " AssociationProcessor* associationStats={}] ---- controlFileData:ControlFileData " source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" |rex " AssociationProcessor* associationStats={}] ---- controlFileData:ControlFileData busDt=(?<busDt>),fileName=(?<fileName>),totalClosingBal=(?<totalClosingBal>)"|table _time  busDt fileName totalClosingBal|sort _time

But I am getting this file name in my statistics "fileName=SETTLEMENT_TRANSFORM_MERGE"  rather I want the one inside Association Stats "SETTLEMENT_TRANSFORM_ASSOCIATION"

Can someone gu

Labels (3)
Tags (1)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust
|rex "fileName=(?<fileName>SETTLEMENT_TRANSFORM_ASSOCIATION)"

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

Try testing you regex in regex101.com to see what it is doing and hopefully figure out what needs to change.

I have made a start for you https://regex101.com/r/Uylo38/1

Hint: * means zero or more of the previous character (or match group) and [ is a special character in regex so would need to be escaped if you want to match with an actual [ in your string.

0 Karma

aditsss
Motivator

@ITWhisperer 

Can you help me here  I need to sow this panel tomorrow

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Your regex statement is doing this:

AssociationProcesso
matches the characters AssociationProcesso literally (case sensitive)
 
r
matches the character r with index 11410 (7216 or 1628) literally (case sensitive)
* matches the previous token between zero and unlimited times, as many times as possible, giving back as needed (greedy)
 
associationStats={}]----controlFileData:ControlFileDatabusDt=
matches the characters associationStats={}] ---- controlFileData:ControlFileData busDt= literally (case sensitive)
Named Capture Group busDt
(?<busDt>)
 — always finds a zero-length match
 
,fileName=
matches the characters ,fileName= literally (case sensitive)
Named Capture Group fileName
(?<fileName>)
 — always finds a zero-length match
 
,totalClosingBal=
matches the characters ,totalClosingBal= literally (case sensitive)
Named Capture Group totalClosingBal
(?<totalClosingBal>)
 — always finds a zero-length match
0 Karma

aditsss
Motivator

@ITWhisperer 

what regex I should use please guide 

I tried with this:

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 sourcetype = "600000304_gg_abs_ipc2" " associationStats={}] ---- controlFileData: ControlFileData " source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" |rex " associationStats={}] ---- controlFileData: ControlFileData " busDt=(?<busDt>),fileName=(?<fileName>),totalClosingBal=(?<totalClosingBal>)"|table _time  busDt fileName totalClosingBal|sort _time

But its taking the file other log also that is why I use AssociationProcessor*

please guide

Below is the screenshot I want to fetch first one

Capture6.PNG

 

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You can still use AssociationProcessor in your search filter, it doesn't have to also be in your regex

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 sourcetype = "600000304_gg_abs_ipc2" " AssociationProcessor* associationStats={}] ---- controlFileData:ControlFileData " source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" 
|rex " busDt=(?<busDt>),fileName=(?<fileName>),totalClosingBal=(?<totalClosingBal>)"
|table _time  busDt fileName totalClosingBal
|sort _time

Now you just need to fix the regex - for example, do the strings actually match up with your events? what characters are you tying to capture in the capture groups?

0 Karma

aditsss
Motivator

@ITWhisperer 

I want filename BusDate and closing balance

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

What pattern would find those characters in the capture groups?

Try doing just the first one until you get that right, then move on to the next one - try this out in regex101.com as it tells you what your pattern is matching against.

0 Karma

aditsss
Motivator

@ITWhisperer 

I tried below query:

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 "AssociationProcessor* associationStats={}] ---- controlFileData: ControlFileData" source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log"|rex " busDate=(?<busDate>),fileName=(?<fileName>),totalClosingBal=(?<totalClosingBal>)"
|table _time busDate fileName totalClosingBal
|sort _time

Getting below result:

File.PNG

This is not the correct result 

Below is the raw log @ITWhisperer its capturing file name as "fileName=SETTLEMENT_TRANSFORM_MERGE" I WANT FILE NAME TO BE THE ONE PRESENT INSIDE 

associationStats={} THAT IS "fileName=SETTLEMENT_TRANSFORM_ASSOCIATION"

2023-07-29 10:39:52.949 [INFO ] [Thread-3] AssociationProcessor - compareTransformStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=19020051, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=07/28/2023, fileName=SETTLEMENT_TRANSFORM_MERGE, totalAchCurrOutstBalAmt=0.0, totalAchBalLastStmtAmt=0.0, totalClosingBal=7.100761644428E10, sourceName=null, version=1, associationStats={}] ---- controlFileData: ControlFileData [fileName=SETTLEMENT_TRANSFORM_ASSOCIATION, busDate=07/28/2023, fileSequenceNum=0, totalBalanceLastStmt=0.0, totalCurrentOutstBal=0.0, totalRecordsWritten=19020051, totalRecords=0, totalClosingBal=7.100761644428E10]

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Your rex is not capturing anything, you have not pattern inside you capture groups for rex to extract against. The value you are seeing for these fields is the value from the index search. You need to modify the rex so that it finds the right place in the log to start the extract from (this is called an anchor), then define what pattern you want to extract into the capture group / field. Look at what regex101.com is telling you is happening for your regex.

0 Karma

aditsss
Motivator

@ITWhisperer 

I am not sure exactly what rex need to be used here could you please guide

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

OK assuming you anchor to the right "fileName=", how would you describe the characters you want to be included in the fileName field?

0 Karma

aditsss
Motivator

@ITWhisperer 

Below file name I want:

fileName=SETTLEMENT_TRANSFORM_ASSOCIATION

AssociationProcessor - compareTransformStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=19020051, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=07/28/2023, fileName=SETTLEMENT_TRANSFORM_MERGE, totalAchCurrOutstBalAmt=0.0, totalAchBalLastStmtAmt=0.0, totalClosingBal=7.100761644428E10, sourceName=null, version=1, associationStats={}] ---- controlFileData: ControlFileData [fileName=SETTLEMENT_TRANSFORM_ASSOCIATION, busDate=07/28/2023, fileSequenceNum=0, totalBalanceLastStmt=0.0, totalCurrentOutstBal=0.0, totalRecordsWritten=19020051, totalRecords=0, totalClosingBal=7.100761644428E10]

I want to display the information inside associationStats={}] ---- controlFileData: ControlFileData [

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
|rex "fileName=(?<fileName>SETTLEMENT_TRANSFORM_ASSOCIATION)"
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...