How to separate key loggers from raw log

[aditsss]

Hi Team,

I am getting raw log as below:

2023-07-22 09:18:19.454 [INFO ] [Thread-3] AssociationProcessor - compareTransformStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=19996779, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=07/21/2023, fileName=SETTLEMENT_TRANSFORM_MERGE, totalAchCurrOutstBalAmt=0.0, totalAchBalLastStmtAmt=0.0, totalClosingBal=8.933513237882E10, sourceName=null, version=1, associationStats={}] ---- controlFileData: ControlFileData [fileName=SETTLEMENT_TRANSFORM_ASSOCIATION, busDate=07/21/2023, fileSequenceNum=0, totalBalanceLastStmt=0.0, totalCurrentOutstBal=0.0, totalRecordsWritten=19996779, totalRecords=0, totalClosingBal=8.933513237882E10]

I want to show each count separately how can we show that:

totalOutputRecords=19996779,
totalClosingBal=8.933513237882E10

How can we create query like this:

index= "abc" sourcetype = "600000304_gg_abs_ipc2" "AssociationProcessor

[ITWhisperer]

Given that you have asked similar questions before and have been shown how to extract information from log events, what have you tried so far for this scenario?

[aditsss]

@ITWhisperer 

I tried but not able to get the correct result can you please guide me here.

[ITWhisperer]

What have you tried?

[aditsss]

@ITWhisperer 

I have Tried with below query:

index= "abc*" sourcetype = "600000304_gg_abs_ipc2" "Post ASSOCIATION" source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" |rex "ASSOCIATION\s+(?<message1>.*)"|table message1 _time

getting below result:

messgae1:

totalInputRecordsCount=19011600, totalOutputRecordsCount=19011598, totalOutstBalFeeAm=8.512726772817E10, nonFinChargeAccounts=17711858, finChargeAccounts=18721170, nonFinCycleAccounts=628, plasticChngAccounts=22298, legalEntityChangeAccounts=0, resv2NonResvAccounts=28, nonresv2ResvAccounts=2694, newAccounts=20663, c2AAccounts=24, acctTermChngCount=155431, excludeAcctCount=0, dailyComputeCount=0, mcaCdChngCount=0, productChngCount=3815

I want to serrate each one of them as I need to show only specific information required like totalInputRecordsCount=19011600, totalOutputRecordsCount=19011598 only these two

@ITWhisperer please guide

[ITWhisperer]

Based on what your rex command currently does, how would you create a new rex command to find the strings you are interested in (the anchors) and extract the values following the anchors?

[aditsss]

@ITWhisperer  Is this possible to get only that two result from that query please guide.

[ITWhisperer]

Here is a guide to how the rex command works

rex - Splunk Documentation

[aditsss]

Hi @ITWhisperer 

I am getting current result as below:

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 sourcetype = "600000304_gg_abs_ipc2" "Post ASSOCIATION" source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" |rex "Post ASSOCIATION\s+(?<message1>.*)"|table message1 _time

How can I just show  totalInputRecordCount along with the count.

totalInputRecordsCount = 19011600

[ITWhisperer]

Explain to me what you think the rex command is doing in your search.

[aditsss]

@ITWhisperer 

This Rex command is just giving me all the log after POST ASSOCIATION

But I want only specific information from that logs

[ITWhisperer]

How exactly does the rex command do that?

[aditsss]

@ITWhisperer 

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 sourcetype = "600000304_gg_abs_ipc2" "Post ASSOCIATION" source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" |rex "Post ASSOCIATION\s+(?<message1>.*)"|table message1 _time

The rex command is giving me all results after POST ASSOCIATION but I only want specific information out of it.

[aditsss]

@ITWhisperer 

Could you please guide me how can I use regex to get that expression

[ITWhisperer]

This is the command you are already using

|rex "Post ASSOCIATION\s+(?<message1>.*)"

You have the _raw log events it is working against

You have the results in the message1 field

You have the documentation for the rex command

You can use use regex101.com as a guide to what the expression is doing and see it working if you paste in your data.

You just need to put a bit of effort into learning what is going on and then try and figure out how to change it to get the new data that you want.

There is a saying about giving fish or teaching how to fish, this is a case of the latter.

[ITWhisperer]

So, if I understand correctly, you are using the rex command without understanding what it does or how you might modify it to get a different result?

[aditsss]

@ITWhisperer 

Can you please help out here to get each of them like InputNumberOf records =189