topic How to separate key loggers from raw log in Dashboards & Visualizations

How to separate key loggers from raw log

aditsss — Fri, 28 Jul 2023 09:44:59 GMT

Hi Team,

I am getting raw log as below:

2023-07-22 09:18:19.454 [INFO ] [Thread-3] AssociationProcessor - compareTransformStatsData : statisticData: StatisticData [selectedDataSet=0, rejectedDataSet=0, totalOutputRecords=19996779, totalInputRecords=0, fileSequenceNum=0, fileHeaderBusDt=null, busDt=07/21/2023, fileName=SETTLEMENT_TRANSFORM_MERGE, totalAchCurrOutstBalAmt=0.0, totalAchBalLastStmtAmt=0.0, totalClosingBal=8.933513237882E10, sourceName=null, version=1, associationStats={}] ---- controlFileData: ControlFileData [fileName=SETTLEMENT_TRANSFORM_ASSOCIATION, busDate=07/21/2023, fileSequenceNum=0, totalBalanceLastStmt=0.0, totalCurrentOutstBal=0.0, totalRecordsWritten=19996779, totalRecords=0, totalClosingBal=8.933513237882E10]

I want to show each count separately how can we show that:

totalOutputRecords=19996779, totalClosingBal=8.933513237882E10

How can we create query like this:

index= "abc" sourcetype = "600000304_gg_abs_ipc2" "AssociationProcessor

Re: How to separate key loggers from raw log

ITWhisperer — Tue, 25 Jul 2023 11:32:55 GMT

Given that you have asked similar questions before and have been shown how to extract information from log events, what have you tried so far for this scenario?

Re: How to separate key loggers from raw log

aditsss — Tue, 25 Jul 2023 12:21:21 GMT

@ITWhisperer

I tried but not able to get the correct result can you please guide me here.

Re: How to separate key loggers from raw log

ITWhisperer — Tue, 25 Jul 2023 12:50:26 GMT

What have you tried?

Re: How to separate key loggers from raw log

aditsss — Thu, 27 Jul 2023 10:43:48 GMT

@ITWhisperer

I have Tried with below query:

index= "abc*" sourcetype = "600000304_gg_abs_ipc2" "Post ASSOCIATION" source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" |rex "ASSOCIATION\s+(?<message1>.*)"|table message1 _time

getting below result:

messgae1:

totalInputRecordsCount=19011600, totalOutputRecordsCount=19011598, totalOutstBalFeeAm=8.512726772817E10, nonFinChargeAccounts=17711858, finChargeAccounts=18721170, nonFinCycleAccounts=628, plasticChngAccounts=22298, legalEntityChangeAccounts=0, resv2NonResvAccounts=28, nonresv2ResvAccounts=2694, newAccounts=20663, c2AAccounts=24, acctTermChngCount=155431, excludeAcctCount=0, dailyComputeCount=0, mcaCdChngCount=0, productChngCount=3815

I want to serrate each one of them as I need to show only specific information required like totalInputRecordsCount=19011600, totalOutputRecordsCount=19011598 only these two

@ITWhisperer please guide

Re: How to separate key loggers from raw log

ITWhisperer — Thu, 27 Jul 2023 10:43:16 GMT

Based on what your rex command currently does, how would you create a new rex command to find the strings you are interested in (the anchors) and extract the values following the anchors?

Re: How to separate key loggers from raw log

aditsss — Thu, 27 Jul 2023 12:10:23 GMT

@ITWhisperer Is this possible to get only that two result from that query please guide.

Re: How to separate key loggers from raw log

ITWhisperer — Thu, 27 Jul 2023 15:30:21 GMT

Here is a guide to how the rex command works

rex - Splunk Documentation

Re: How to separate key loggers from raw log

aditsss — Thu, 27 Jul 2023 15:58:07 GMT

Hi @ITWhisperer

I am getting current result as below:

index="600000304_d_gridgain_idx*" sourcetype=600000304_gg_abs_ipc2 sourcetype = "600000304_gg_abs_ipc2" "Post ASSOCIATION" source="/amex/app/gfp-settlement-transform/logs/gfp-settlement-transform.log" |rex "Post ASSOCIATION\s+(?<message1>.*)"|table message1 _time

How can I just show totalInputRecordCount along with the count.

totalInputRecordsCount = 19011600

Re: How to separate key loggers from raw log

ITWhisperer — Thu, 27 Jul 2023 17:13:41 GMT

Explain to me what you think the rex command is doing in your search.

Re: How to separate key loggers from raw log

aditsss — Thu, 27 Jul 2023 17:51:38 GMT

@ITWhisperer

This Rex command is just giving me all the log after POST ASSOCIATION

But I want only specific information from that logs

Re: How to separate key loggers from raw log

ITWhisperer — Thu, 27 Jul 2023 17:59:49 GMT

How exactly does the rex command do that?

Re: How to separate key loggers from raw log

aditsss — Thu, 27 Jul 2023 18:35:18 GMT

@ITWhisperer

The rex command is giving me all results after POST ASSOCIATION but I only want specific information out of it.

Re: How to separate key loggers from raw log

ITWhisperer — Thu, 27 Jul 2023 21:47:17 GMT

So, if I understand correctly, you are using the rex command without understanding what it does or how you might modify it to get a different result?

Re: How to separate key loggers from raw log

aditsss — Fri, 28 Jul 2023 05:38:51 GMT

@ITWhisperer

Could you please guide me how can I use regex to get that expression

Re: How to separate key loggers from raw log

aditsss — Fri, 28 Jul 2023 05:49:06 GMT

@ITWhisperer

Can you please help out here to get each of them like InputNumberOf records =189

Re: How to separate key loggers from raw log

ITWhisperer — Fri, 28 Jul 2023 09:22:54 GMT

This is the command you are already using

|rex "Post ASSOCIATION\s+(?<message1>.*)"

You have the _raw log events it is working against

You have the results in the message1 field

You have the documentation for the rex command

You can use use regex101.com as a guide to what the expression is doing and see it working if you paste in your data.

You just need to put a bit of effort into learning what is going on and then try and figure out how to change it to get the new data that you want.

There is a saying about giving fish or teaching how to fish, this is a case of the latter.