How to fetch the values from raw logs

[aditsss]

Hi All,

Below is my raw log and I want to fetch the highlighted value from it:

2023-08-08 10:25:48.407 [INFO ] [Thread-3] CollateralProcessor - compareCollateralStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=0, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=08/06/2023, fileName=SETTLEMENT_TRANSFORM_COLLATERAL_LENDING, totalAchCurrOutstBalAmt=2.722379487286E10, totalAchBalLastStmtAmt=2.722379487286E10, totalClosingBal=2.722379487286E10, sourceName=null, version=0, associationStats={}] with collateralSum 2.722379487286E10 openingBal 2.73215647389E10 ageBalTot 2.722379487286E10 busDt 08/07/2023 with prevStatisticData null

Below is my query but I am not able to fetch that:

index="abc*" sourcetype =600000304_gg_abs_ipc2 source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" "CollateralProcessor - compareCollateralStatsData"|rex "CollateralProcessor - compareCollateralStatsData busDt=(?<busDt>),fileName=(?<fileName>),collateralSum =(?<collateralSum>)"|table busDt fileName collateralSum | sort busDt

Can someone guide me how I can fetch highlight

[bowesmana]

@aditsss when posting code snippets, please use the code tag </> to make your code easier to read. Also it helps to split the command pipes on to separate lines for ease of reading - you can do this with the Ctrl-\ character when in the Splunk UI SPL editor

[isoutamo]

Hi

for me it seems to work as @smurf suggested with your test data. See below

| makeresults
| eval _raw = "2023-08-08 10:25:48.407 [INFO ] [Thread-3] CollateralProcessor - compareCollateralStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=0, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=08/06/2023, fileName=SETTLEMENT_TRANSFORM_COLLATERAL_LENDING, totalAchCurrOutstBalAmt=2.722379487286E10, totalAchBalLastStmtAmt=2.722379487286E10, totalClosingBal=2.722379487286E10, sourceName=null, version=0, associationStats={}] with collateralSum 2.722379487286E10 openingBal 2.73215647389E10 ageBalTot 2.722379487286E10 busDt 08/07/2023 with prevStatisticData null 
2023-08-09 10:31:57.834 [INFO ] [Thread-3] CollateralProcessor - compareCollateralStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=0, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=08/07/2023, fileName=SETTLEMENT_TRANSFORM_COLLATERAL_LENDING, totalAchCurrOutstBalAmt=2.71092692285E10, totalAchBalLastStmtAmt=2.71092692285E10, totalClosingBal=2.71092692285E10, sourceName=null, version=0, associationStats={}] with collateralSum 2.71092692285E10 openingBal 2.722379487286E10 ageBalTot 2.71092692285E10 busDt 08/08/2023 with prevStatisticData null"
| multikv noheader=t
```previous make test events```
| rex "busDt=(?<busDt>.*?),\sfileName=(?<fileName>.+?),.*collateralSum\s(?<collateralSum>[\d|\.|E]+)\sopeningBal\s(?<openingBal>[\d|\.|E]+)\sageBalTot\s(?<ageBalTot>[\d|\.|E]+)"
| table busDt fileName collateralSum openingBal ageBalTot

This shows those two events with required fields.

Works even you change that regex to 

| rex "busDt=(?<busDt>.*?),\sfileName=(?<fileName>.+?),.*collateralSum\s(?<collateralSum>[\d\.E]+)\sopeningBal\s(?<openingBal>[\d\.E]+)\sageBalTot\s(?<ageBalTot>[\d\.E]+)"

You shouldn't use "|" as a separator inside "[]".

r. Ismo 

[smurf]

Hi,

your regex seems wrong, so it does not extract anything. I recommend using something like regex101 to test your regex.

Try replacing it with this (this should work with your sample event):

| rex "busDt=(?<busDt>.*?),\sfileName=(?<fileName>.+?),.*collateralSum\s(?<collateralSum>[\d|\.|E]+)\sopeningBal\s(?<openingBal>[\d|\.|E]+)\sageBalTot\s(?<ageBalTot>[\d|\.|E]+)"

Hope this helps.

smurf

[aditsss]

@smurf 

I used this:

index="abc*" sourcetype =600000304_gg_abs_ipc2 source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" "CollateralProcessor - compareCollateralStatsData"| rex "busDt=(?<busDt>.*?),\sfileName=(?<fileName>.+?),.*collateralSum\s(?<collateralSum>[\d|\.|E]+)\sopeningBal\s(?<openingBal>[\d|\.|E]+)\sageBalTot\s(?<ageBalTot>[\d|\.|E]+)"|table busDt fileName collateralSum openingBal

But not able to see openingBal and collateralSum

Below screenshot:

raw logs:

2023-08-09 10:31:57.834 [INFO ] [Thread-3] CollateralProcessor - compareCollateralStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=0, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=08/07/2023, fileName=SETTLEMENT_TRANSFORM_COLLATERAL_LENDING, totalAchCurrOutstBalAmt=2.71092692285E10, totalAchBalLastStmtAmt=2.71092692285E10, totalClosingBal=2.71092692285E10, sourceName=null, version=0, associationStats={}]

with collateralSum 2.71092692285E10 openingBal 2.722379487286E10 ageBalTot 2.71092692285E10 busDt 08/08/2023 with prevStatisticData null