How to fetch the variable with the same name from raw logs?

[aditsss]

Hi Team,

My raw logs are below:

2023-08-04 10:06:12.750 [INFO ] [Thread-3] AssociationProcessor - compareTransformStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=17897259, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=08/03/2023, fileName=SETTLEMENT_TRANSFORM_MERGE, totalAchCurrOutstBalAmt=0.0, totalAchBalLastStmtAmt=0.0, totalClosingBal=8.787189909105E10, sourceName=null, version=1, associationStats={}] ---- controlFileData: ControlFileData [fileName=SETTLEMENT_TRANSFORM_ASSOCIATION, busDate=08/03/2023, fileSequenceNum=0, totalBalanceLastStmt=0.0, totalCurrentOutstBal=0.0, totalRecordsWritten=17897259, totalRecords=0, totalClosingBal=8.787189909105E10]

I want to fetch both files my current query is:

index="abc*" sourcetype =600000304_gg_abs_ipc2 " AssociationProcessor - compareTransformStatsData : statisticData: StatisticData" source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" 
| rex " AssociationProcessor - compareTransformStatsData : statisticData: StatisticData totalOutputRecords=(?<totalOutputRecords>),busDt=(?<busDt>),fileName=(?<fileName>),totalClosingBal=(?<totalClosingBal>)" 
| eval TotalClosingBalance=tonumber(mvindex(split(totalClosingBal,"E"),0)) * pow(10,tonumber(mvindex(split(totalClosingBal,"E"),1))) 
| table busDt fileName totalOutputRecords TotalClosingBalance 
| sort busDt 
| appendcols 
    [ search index="600000304_d_gridgain_idx*" sourcetype =600000304_gg_abs_ipc2 " AssociationProcessor* associationStats={}] ---- controlFileData: ControlFileData" source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" 
    | rex " AssociationProcessor* - associationStats={}] ---- controlFileData: ControlFileData ,busDate=(?<busDate>),fileSequenceNum=(?<fileSequenceNum>),totalRecordsWritten=(?<totalRecordsWritten>),totalRecords=(?<totalRecords>),totalClosingBal=(?<totalClosingBal>)" 
    | rex "fileName=(?<fileName>SETTLEMENT_TRANSFORM_ASSOCIATION)" 
    | eval TotalClosingBalance=tonumber(mvindex(split(totalClosingBal,"E"),0)) * pow(10,tonumber(mvindex(split(totalClosingBal,"E"),1))) 
    | table busDate busDate fileName totalRecordsWritten TotalClosingBalance] 
| sort busDt

But I am getting file name as SETTLEMENT_TRANSFORM_MERGE But I want both SETTLEMENT_TRANSFORM_MERGE and SETTLEMENT_TRANSFORM_ASSOCIATION both

please guide

[richgalloway]

The rex commands need to have expressions in the capture groups.  Also, and perhaps more important, the regular expression used in rex must match the data before Splunk will extract any fields.  In the sample event, each key=value pair is separated by a comma and space, but the regex only looks for a comma.  Additionally, the regexes do not account for intervening key=value pairs that are not extracted - these must still match to extract fields.

The final reason why you're getting only one fileName value is the appendcols command is being asked to add column names that already exist (busDate, fileName, TotalClosingBalance).  It won't do that.  It will either keep the existing fields or replace them, but it won't duplicate them.  See https://docs.splunk.com/Documentation/Splunk/9.1.0/SearchReference/Appendcols#Optional_arguments

See this run-anywhere example of a query that works, but not produce the exact desired results.

| makeresults | eval _raw="2023-08-04 10:06:12.750 [INFO ] [Thread-3] AssociationProcessor - compareTransformStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=17897259, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=08/03/2023, fileName=SETTLEMENT_TRANSFORM_MERGE, totalAchCurrOutstBalAmt=0.0, totalAchBalLastStmtAmt=0.0, totalClosingBal=8.787189909105E10, sourceName=null, version=1, associationStats={}] ---- controlFileData: ControlFileData [fileName=SETTLEMENT_TRANSFORM_ASSOCIATION, busDate=08/03/2023, fileSequenceNum=0, totalBalanceLastStmt=0.0, totalCurrentOutstBal=0.0, totalRecordsWritten=17897259, totalRecords=0, totalClosingBal=8.787189909105E10]"
| rex " AssociationProcessor - compareTransformStatsData : statisticData: StatisticData .+? totalOutputRecords=(?<totalOutputRecords>[^,]+),.*?busDt=(?<busDt>[^,]+), fileName=(?<fileName>[^,]+),.*?totalClosingBal=(?<totalClosingBal>[^,]+)" 
| eval TotalClosingBalance=tonumber(mvindex(split(totalClosingBal,"E"),0)) * pow(10,tonumber(mvindex(split(totalClosingBal,"E"),1))) 
| table busDt fileName totalOutputRecords TotalClosingBalance 
| sort busDt 
| appendcols 
    [  | makeresults | eval _raw="2023-08-04 10:06:12.750 [INFO ] [Thread-3] AssociationProcessor - compareTransformStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=17897259, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=08/03/2023, fileName=SETTLEMENT_TRANSFORM_MERGE, totalAchCurrOutstBalAmt=0.0, totalAchBalLastStmtAmt=0.0, totalClosingBal=8.787189909105E10, sourceName=null, version=1, associationStats={}] ---- controlFileData: ControlFileData [fileName=SETTLEMENT_TRANSFORM_ASSOCIATION, busDate=08/03/2023, fileSequenceNum=0, totalBalanceLastStmt=0.0, totalCurrentOutstBal=0.0, totalRecordsWritten=17897259, totalRecords=0, totalClosingBal=8.787189909105E10]" 
    | rex " associationStats={}] ---- controlFileData: ControlFileData \[fileName=(?<fileName2>[^,]+), busDate=(?<busDate2>[^,]+), fileSequenceNum=(?<fileSequenceNum>[^,]+),.*?totalRecordsWritten=(?<totalRecordsWritten>[^,]+), totalRecords=(?<totalRecords>[^,]+), totalClosingBal=(?<totalClosingBal2>[^\]]+)" 
    | eval TotalClosingBalance=tonumber(mvindex(split(totalClosingBal,"E"),0)) * pow(10,tonumber(mvindex(split(totalClosingBal,"E"),1))) 
    | table busDate2 fileName2 totalRecordsWritten TotalClosingBalance2]
| sort busDt