topic Re: How to extract some fields from a part json part text log in Splunk in Splunk Search

How to extract some fields from a part json part text log in Splunk

bhartiya007 — Wed, 17 Feb 2021 11:25:29 GMT

I am fairly new to splunk and still learning. I have a splunk event which is a mix of some texts and json in between. (This isn't the complete log)

2021-02-14 00:00:03,596 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO com.growl.hdt.dmt.DQ.bapm.RetrieveDataFromDQ - Total Application assets -> 1692 2021-02-14 00:00:03,596 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO com.growl.hdt.dmt.DQ.bapm.CommonUtils - {"Header":{"AppId":"AD00006933","Type":"Inbound","RecId":"416c627c-41a7-428e-a871-5317c4842fe5","StartTS":"2021-02-14T05:00Z","Ver":"2.0.0"},"Application":{"APP_OS":"Linux 3.10.0-1160.11.1.el7.x86_64","APP_Runtime":"Java 1.8.0_282","APP_AppName":"DQ-bapm-Integration","APP_AppVersion":"1.0.0","Host":"zebra.cdc.growl.com/10.102.180.53","Channel":"Other"},"Service":{"Key":"DQ2bapm","URL":"https://growl-test.DQ.com/rest/2.0/assets?limit=1000&offset=1000&typeId=00000000-0000-0000-0000-000000031302&communityId=595b27d3-ff42-45e4-8dc7-0172f7d82693&domainId=2c8b39ea-0d7f-445f-acc2-a1fb3a9a12db&statusId=00000000-0000-0000-0000-000000005009","CallType":"REST","Operation":"GET"},"Results":{"Elapsed":"0","Message":"Invoking DQ REST API","TraceLevel":"DEBUG"},"Security":{"Vendor":"growl"}} 2021-02-14 00:00:03,795 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO com.growl.hdt.dmt.DQ.bapm.RetrieveDataFromDQ - Total Application assets -> 1692 2021-02-14 00:00:03,795 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO com.growl.hdt.dmt.DQ.bapm.RetrieveDataFromDQ - Total Application assets in appAssetList-> 1692 2021-02-14 00:00:04,499 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO com.growl.hdt.dmt.DQ.bapm.ComparebapmDQRecords - List of Applications in DQ to be marked "Obsolete in bapm": [AD00007661, AD00007470, AD00007539, AD00007549, AD00007643] 2021-02-14 00:00:04,499 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO com.growl.hdt.dmt.DQ.bapm.ComparebapmDQRecords - ## Total Application count from bapm ##1696 2021-02-14 00:00:04,499 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO com.growl.hdt.dmt.DQ.bapm.ComparebapmDQRecords - ## Total Application Asset in DQ ##1692 2021-02-14 00:00:04,499 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO com.growl.hdt.dmt.DQ.bapm.ComparebapmDQRecords - ## No of Application to Obsolete in DQ ##5

How can I extract the below :

List of Applications in DQ to be marked "Obsolete in bapm": 
[AD00007661, AD00007470, AD00007539, AD00007549, AD00007643]
Total Application count from bapm ##1696
Total Application Asset in DQ ##1692
No of Application to Obsolete in DQ ##5

Re: How to extract some fields from a part json part text log in Splunk

ITWhisperer — Wed, 17 Feb 2021 11:34:59 GMT

Are these separate events?

Can you just not add something to the search to just get the events you want

("List of Applications in DQ to be marked" OR "Total Application count from bapm" OR "Total Application Asset in DQ" OR "No of Application to Obsolete in DQ")

Re: How to extract some fields from a part json part text log in Splunk

bhartiya007 — Wed, 17 Feb 2021 12:07:43 GMT

@ITWhisperer

No the info is coming in the same event.
I tried the below but I am getting the full events instead of the fields and their respective values.
index=hdt sourcetype=hdt_dg_logs "List of Applications in DQ to be marked" OR "Total Application count from bapm" OR "Total Application Asset in DQ" OR "No of Application to Obsolete in DQ"

I want something in the tabular format with the only the above counts.
Sorry If my requirement is not correct I am pretty new to these.

Re: How to extract some fields from a part json part text log in Splunk

bhartiya007 — Wed, 17 Feb 2021 11:57:40 GMT

I need to generate an alert with the format like below

List of Applications in DQ to be marked "Obsolete in bapm": 
[AD00007661, AD00007470, AD00007539, AD00007549, AD00007643]
Total Application count from bapm=1696
Total Application Asset in DQ =1692
No of Application to Obsolete in DQ =5

Re: How to extract some fields from a part json part text log in Splunk

ITWhisperer — Wed, 17 Feb 2021 12:11:15 GMT

It is probably worth splitting these lines into separate events? If you can't do this when ingesting the events, you could try

| rex max_match=0 "(?<line>.+\n)" | mvexpand line | fields - _raw

Re: How to extract some fields from a part json part text log in Splunk

bhartiya007 — Wed, 17 Feb 2021 12:43:44 GMT

@ITWhisperer
When I run this i get two events where these values exists.

index=hdt sourcetype=hdt_dg_logs "List of Applications in DQ to be marked" OR "Total Application count from bapm" OR "Total Application Asset in DQ" OR "No of Application to Obsolete in DQ"

Do you want me to run this entire query?
index=hdt sourcetype=hdt_dg_logs "List of Applications in DQ to be marked" OR "Total Application count from bapm" OR "Total Application Asset in DQ" OR "No of Application to Obsolete in DQ" | rex max_match=0 "(?<line>.+\n)"
| mvexpand line
| fields - _raw

If not , I probably didn't get you here. can you please provide me the query to work with?

Re: How to extract some fields from a part json part text log in Splunk

ITWhisperer — Wed, 17 Feb 2021 13:29:57 GMT

index=hdt sourcetype=hdt_dg_logs | rex max_match=0 "(?<line>.+\n)" | mvexpand line | fields - _raw | search "List of Applications in DQ to be marked" OR "Total Application count from bapm" OR "Total Application Asset in DQ" OR "No of Application to Obsolete in DQ"

If you then want to get rid of the beginning part of the line you could do something like

| eval line=mvindex(split(line,"-"),3)

Re: How to extract some fields from a part json part text log in Splunk

bhartiya007 — Wed, 17 Feb 2021 14:36:53 GMT

@ITWhisperer
This isn't bringing any data/event.

Re: How to extract some fields from a part json part text log in Splunk

ITWhisperer — Wed, 17 Feb 2021 14:43:01 GMT

Does the first part at least break it down into separate events?

index=hdt sourcetype=hdt_dg_logs | rex max_match=0 "(?<line>.+\n)" | mvexpand line | fields - _raw

Re: How to extract some fields from a part json part text log in Splunk

bhartiya007 — Wed, 17 Feb 2021 14:49:17 GMT

@ITWhisperer
The number of events have increased from 13 to 359 but the only thing i can see are the selected fields in those events index source and sourcetype

Re: How to extract some fields from a part json part text log in Splunk

ITWhisperer — Wed, 17 Feb 2021 14:56:37 GMT

Sorry - try this

index=hdt sourcetype=hdt_dg_logs | rex max_match=0 "(?<line>.+\n)" | mvexpand line | eval _raw=line | fields - line | search "List of Applications in DQ to be marked" OR "Total Application count from bapm" OR "Total Application Asset in DQ" OR "No of Application to Obsolete in DQ"

Re: How to extract some fields from a part json part text log in Splunk

bhartiya007 — Wed, 17 Feb 2021 16:17:18 GMT

@ITWhisperer Thank you for helping me out here. Much Appreciated.
So the latest query brought the below two events

021-02-14 00:00:04,499 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO
com.growl.hdt.dmt.DQ.bapm.ComparebapmDQRecords - List of Applications in DQ to be marked "Obsolete in bapm":
2021-02-14 00:00:03,274 [[bapm2DQ].bapmprojectFlow.stage1.04] INFO
com.growl.hdt.dmt.DQ.bapm.ComparebapmDQRecords - List of Applications in DQ to be marked "Obsolete in bapm":

But It didn't bring the other search items.

Also, in the above the application names are missing i.e it didn't show the entire line as below:
List of Applications in DQ to be marked "Obsolete in bapm":[AD00007661, AD00007470, AD00007539, AD00007549, AD00007643]