Splunk Search

How to extract some fields from a part json part text log in Splunk

bhartiya007
Loves-to-Learn Lots

I am fairly new to splunk and still learning. I have a splunk event which is a mix of some texts and json in between. (This isn't the complete log)

 

 

 

2021-02-14 00:00:03,596 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO  com.growl.hdt.dmt.DQ.bapm.RetrieveDataFromDQ - Total Application assets -> 1692
2021-02-14 00:00:03,596 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO  com.growl.hdt.dmt.DQ.bapm.CommonUtils - {"Header":{"AppId":"AD00006933","Type":"Inbound","RecId":"416c627c-41a7-428e-a871-5317c4842fe5","StartTS":"2021-02-14T05:00Z","Ver":"2.0.0"},"Application":{"APP_OS":"Linux 3.10.0-1160.11.1.el7.x86_64","APP_Runtime":"Java 1.8.0_282","APP_AppName":"DQ-bapm-Integration","APP_AppVersion":"1.0.0","Host":"zebra.cdc.growl.com/10.102.180.53","Channel":"Other"},"Service":{"Key":"DQ2bapm","URL":"https://growl-test.DQ.com/rest/2.0/assets?limit=1000&offset=1000&typeId=00000000-0000-0000-0000-000000031302&communityId=595b27d3-ff42-45e4-8dc7-0172f7d82693&domainId=2c8b39ea-0d7f-445f-acc2-a1fb3a9a12db&statusId=00000000-0000-0000-0000-000000005009","CallType":"REST","Operation":"GET"},"Results":{"Elapsed":"0","Message":"Invoking DQ REST API","TraceLevel":"DEBUG"},"Security":{"Vendor":"growl"}}
2021-02-14 00:00:03,795 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO  com.growl.hdt.dmt.DQ.bapm.RetrieveDataFromDQ - Total Application assets -> 1692
2021-02-14 00:00:03,795 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO  com.growl.hdt.dmt.DQ.bapm.RetrieveDataFromDQ - Total Application assets in appAssetList-> 1692
2021-02-14 00:00:04,499 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO  com.growl.hdt.dmt.DQ.bapm.ComparebapmDQRecords - List of Applications in DQ to be marked "Obsolete in bapm": 
[AD00007661, AD00007470, AD00007539, AD00007549, AD00007643]
2021-02-14 00:00:04,499 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO  com.growl.hdt.dmt.DQ.bapm.ComparebapmDQRecords - ## Total Application count from bapm ##1696
2021-02-14 00:00:04,499 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO  com.growl.hdt.dmt.DQ.bapm.ComparebapmDQRecords - ## Total Application Asset in DQ ##1692
2021-02-14 00:00:04,499 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO  com.growl.hdt.dmt.DQ.bapm.ComparebapmDQRecords - ## No of Application to Obsolete in DQ ##5

 

 

 

How can I extract the below :

List of Applications in DQ to be marked "Obsolete in bapm": 
[AD00007661, AD00007470, AD00007539, AD00007549, AD00007643]
Total Application count from bapm ##1696
Total Application Asset in DQ ##1692
No of Application to Obsolete in DQ ##5
Labels (2)
Tags (1)
0 Karma

bhartiya007
Loves-to-Learn Lots

I need to generate an alert with the format like below

List of Applications in DQ to be marked "Obsolete in bapm": 
[AD00007661, AD00007470, AD00007539, AD00007549, AD00007643]
Total Application count from bapm=1696
Total Application Asset in DQ =1692
No of Application to Obsolete in DQ =5
0 Karma

bhartiya007
Loves-to-Learn Lots

@ITWhisperer 

No the info is coming in the same event.
I tried the below  but I am getting the full events instead of the fields and their respective values.
index=hdt sourcetype=hdt_dg_logs "List of Applications in DQ to be marked" OR "Total Application count from bapm" OR "Total Application Asset in DQ" OR "No of Application to Obsolete in DQ"

I want something in the tabular format with the only the above counts.
Sorry If my requirement is not correct I am pretty new to these.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

It is probably worth splitting these lines into separate events? If you can't do this when ingesting the events, you could try

| rex max_match=0 "(?<line>.+\n)"
| mvexpand line
| fields - _raw
0 Karma

bhartiya007
Loves-to-Learn Lots

@ITWhisperer 
When I run this i get two events where these values exists.

index=hdt sourcetype=hdt_dg_logs "List of Applications in DQ to be marked" OR "Total Application count from bapm" OR "Total Application Asset in DQ" OR "No of Application to Obsolete in DQ"

Do you want me to run this entire query? 
index=hdt sourcetype=hdt_dg_logs "List of Applications in DQ to be marked" OR "Total Application count from bapm" OR "Total Application Asset in DQ" OR "No of Application to Obsolete in DQ"  | rex max_match=0 "(?<line>.+\n)"
| mvexpand line
| fields - _raw

If not , I probably didn't get you here. can you please provide me the query to work with?

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
index=hdt sourcetype=hdt_dg_logs 
| rex max_match=0 "(?<line>.+\n)"
| mvexpand line
| fields - _raw
| search "List of Applications in DQ to be marked" OR "Total Application count from bapm" OR "Total Application Asset in DQ" OR "No of Application to Obsolete in DQ"

If you then want to get rid of the beginning part of the line you could do something like

| eval line=mvindex(split(line,"-"),3)

 

0 Karma

bhartiya007
Loves-to-Learn Lots

@ITWhisperer 
This isn't bringing any data/event.

index=hdt sourcetype=hdt_dg_logs | rex max_match=0 "(?<line>.+\n)" | mvexpand line | fields - _raw | search "List of Applications in DQ to be marked" OR "Total Application count from bapm" OR "Total Application Asset in DQ" OR "No of Application to Obsolete in DQ"

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Does the first part at least break it down into separate events?

index=hdt sourcetype=hdt_dg_logs 
| rex max_match=0 "(?<line>.+\n)"
| mvexpand line
| fields - _raw
0 Karma

bhartiya007
Loves-to-Learn Lots

@ITWhisperer 
The number of events have increased from 13 to 359 but the only thing i can see are the selected fields in those events index source and sourcetype

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Sorry - try this

index=hdt sourcetype=hdt_dg_logs 
| rex max_match=0 "(?<line>.+\n)"
| mvexpand line
| eval _raw=line
| fields - line
| search "List of Applications in DQ to be marked" OR "Total Application count from bapm" OR "Total Application Asset in DQ" OR "No of Application to Obsolete in DQ"
0 Karma

bhartiya007
Loves-to-Learn Lots

@ITWhisperer  Thank you for helping me out here. Much Appreciated.
So the latest query brought the below two events


021-02-14 00:00:04,499 [[bapm2DQ].bapmprojectFlow.stage1.02] INFO
com.growl.hdt.dmt.DQ.bapm.ComparebapmDQRecords - List of Applications in DQ to be marked "Obsolete in bapm":
2021-02-14 00:00:03,274 [[bapm2DQ].bapmprojectFlow.stage1.04] INFO
com.growl.hdt.dmt.DQ.bapm.ComparebapmDQRecords - List of Applications in DQ to be marked "Obsolete in bapm":

But It didn't bring the other search items.

Also, in the above the application names are missing i.e it didn't show the entire line as below:
List of Applications in DQ to be marked "Obsolete in bapm":[AD00007661, AD00007470, AD00007539, AD00007549, AD00007643]

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Are these separate events?

Can you just not add something to the search to just get the events you want

("List of Applications in DQ to be marked" OR "Total Application count from bapm" OR "Total Application Asset in DQ" OR "No of Application to Obsolete in DQ")
0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

&#x1f342; Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...