topic Re: Splunk querying nested log in Splunk Search

Splunk querying nested log

shwetamis — Thu, 21 Nov 2019 15:02:05 GMT

I have a log below and I want to get the value of Description under :- Calling Checklist1003
How do I do that ??

Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 09:21:53.297 Fault type: Application Severity: Info
11/21/2019 09:21:53.297 Description: This is a resubmission of a case that was underwritten using the
11/21/2019 09:21:53.297 UW_10.30 KB engine

11/21/2019 09:21:53.297

11/21/2019 09:21:53.297 UWROUTER service will be used for underwriting
11/21/2019 09:21:53.297 ----------------------------------------------------------------
11/21/2019 09:21:53.297 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 09:21:53.297 Fault type: Application Severity: Info
11/21/2019 09:21:53.297 Description: This case will be underwritten using UWROUTER 1.0

11/21/2019 09:21:53.297

11/21/2019 09:21:53.297 Calling Checklist1003
11/21/2019 09:21:53.345 ----------------------------------------------------------------
11/21/2019 09:21:53.345 Message type: Code: 118310 dec, 1ce26 hex
11/21/2019 09:21:53.345 Fault type: Undefined Severity: Undefined
11/21/2019 09:21:53.345 Description: Hired From Date is missing for secondary employment for

11/21/2019 09:21:53.345 applicant .

11/21/2019 09:21:53.345

11/21/2019 09:21:53.358 -----------------------------------------

Re: Splunk querying nested log

dmarling — Thu, 21 Nov 2019 17:23:43 GMT

To confirm, is that one log event or multiple log events in your example?

Re: Splunk querying nested log

shwetamis — Thu, 21 Nov 2019 17:30:54 GMT

One log event

Re: Splunk querying nested log

dmarling — Thu, 21 Nov 2019 17:41:33 GMT

Here's how I would do it. This assumes that the description will never be more than two lines long. Here's a run anywhere example using the data you provided in the initial post:

| makeresults count=1
| eval data="Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 09:21:53.297 Fault type: Application Severity: Info
11/21/2019 09:21:53.297 Description: This is a resubmission of a case that was underwritten using the
11/21/2019 09:21:53.297 UW_10.30 KB engine
11/21/2019 09:21:53.297
11/21/2019 09:21:53.297 UWROUTER service will be used for underwriting
11/21/2019 09:21:53.297 ----------------------------------------------------------------
11/21/2019 09:21:53.297 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 09:21:53.297 Fault type: Application Severity: Info
11/21/2019 09:21:53.297 Description: This case will be underwritten using UWROUTER 1.0
11/21/2019 09:21:53.297
11/21/2019 09:21:53.297 Calling Checklist1003
11/21/2019 09:21:53.345 ----------------------------------------------------------------
11/21/2019 09:21:53.345 Message type: Code: 118310 dec, 1ce26 hex
11/21/2019 09:21:53.345 Fault type: Undefined Severity: Undefined
11/21/2019 09:21:53.345 Description: Hired From Date is missing for secondary employment for
11/21/2019 09:21:53.345 applicant .
11/21/2019 09:21:53.345
11/21/2019 09:21:53.358 -----------------------------------------"
| rex field=data mode=sed "s/([\n\r])\d{2}\/\d{2}\/\d{4} \d{2}:\d{2}:\d{2}.\d{3}/\1/g"
| rex field=data max_match=2 "Description: (?<Description>[^\n]+\n[^\n]+)"
| eval Description=mvindex(Description, -1)

The last three lines of that are applicable to you except you would remove the field=data and replace it with field=_raw. This removes the date/time stamps from the message and grab everything after each Description plus 1 line with the rex and then only returns the last Description in the event with the eval.

Re: Splunk querying nested log

shwetamis — Thu, 21 Nov 2019 17:47:09 GMT

Thank you, this is very good. But I have just pasted a part of the log just to show an example of how my data looks, the field Description is repeated about 7-8times in the log ( log is pretty big), so under eval data do I need to paste the whole log ?

Re: Splunk querying nested log

dmarling — Thu, 21 Nov 2019 17:51:06 GMT

If you could paste a whole log event as an example (with anything sensitive fuzzed out) and confirm if you want all of the descriptions or only the ones after some specific pattern. If it's always after calling checklist1003 I can write a rex for that, if it's something else I would need to know that.

Re: Splunk querying nested log

shwetamis — Wed, 30 Sep 2020 03:03:20 GMT

11/21/2019 12:52:49.929 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:49.929 Fault type: Application Severity: Info
11/21/2019 12:52:49.929 Description: Verification Successful

11/21/2019 12:52:49.929

11/21/2019 12:52:49.929 ----------------------------------------------------------------
11/21/2019 12:52:49.929 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:49.929 Fault type: Application Severity: Info
11/21/2019 12:52:49.929 Description: No errors were detected

11/21/2019 12:52:49.929

11/21/2019 12:52:49.929 Performing a CaseInit
11/21/2019 12:52:49.929 LOAN_APPLICATION found. Send to store as Original Mismo 1003!
11/21/2019 12:52:50.604 Validating reply from CaseInit
11/21/2019 12:52:50.604 ----------------------------------------------------------------
11/21/2019 12:52:50.604 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:50.604 Fault type: Application Severity: Info
11/21/2019 12:52:50.604 Description: CaseInit completed successfully

11/21/2019 12:52:50.604

11/21/2019 12:52:50.604 Inst ID set for EH-020 [12121231]
11/21/2019 12:52:50.604 Casefile ID [xxxxxxx]
11/21/2019 12:52:50.604 lenderCaseNo [yyyyyyy]
11/21/2019 12:52:50.604 OperStatus_in_mp_casefile_set is not defined.
11/21/2019 12:52:50.604 Calling Credential service to get traits
11/21/2019 12:52:50.624 ----------------------------------------------------------------
11/21/2019 12:52:50.624 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:50.624 Fault type: Application Severity: Info
11/21/2019 12:52:50.624 Description: InstTraitRequest completed successfully

11/21/2019 12:52:50.624

11/21/2019 12:52:50.625 Call RESTBridge for conventional loan with GUID [abbababsbasasas],
11/21/2019 12:52:50.625 INST_INFO: 12121231|somecompany
11/21/2019 12:52:50.625

11/21/2019 12:52:50.625 Begin RESTBridgeSubsystem::callRESTBridge() ...
11/21/2019 12:52:50.625 BASE_GUID for VDVS [abbababsbasasas]
11/21/2019 12:52:50.625 Begin RESTBridgeSubsystem::sendRESTBridgeClientRequest...
11/21/2019 12:52:50.625 Call RESTBridgeProxy.sendRESTBridgeRequest:
11/21/2019 12:52:50.625 Svcname= RESTBridge
11/21/2019 12:52:50.625 Major= 1
11/21/2019 12:52:50.625 Minor= 0
11/21/2019 12:52:50.625 Command= RESTBridge
11/21/2019 12:52:50.628 End RESTBridgeSubsystem::sendRESTBridgeClientRequest
11/21/2019 12:52:50.628 End RESTBridgeSubsystem::callRESTBridge()
11/21/2019 12:52:50.628 SBAWF13Service_Impl:Underwrite:BESTFIT_PRODUCT_SPECIFICATIONS file attachment is not found. This is NOT a multi-product case
11/21/2019 12:52:50.629 Determine ProdAlt case:
11/21/2019 12:52:50.629 ProdAlt Flag [OFF]
11/21/2019 12:52:50.629 LoanType [01]
11/21/2019 12:52:50.629 prodAltTrait [0]
11/21/2019 12:52:50.629 This is NOT a Product alternative case!
11/21/2019 12:52:50.629 CONVERSION_TARGET: RES,HTML
11/21/2019 12:52:50.629 User format Request: XML,HTML
11/21/2019 12:52:50.629 Determining the underwriting service to be used
11/21/2019 12:52:50.629 Set to use Default engine UWROUTER,1,0 for underwriting
11/21/2019 12:52:50.629 Underwrite::determineUWService::multiProductCase =0
11/21/2019 12:52:50.629 This is not a multi-product case.
11/21/2019 12:52:50.629 ----------------------------------------------------------------
11/21/2019 12:52:50.629 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:50.629 Fault type: Application Severity: Info
11/21/2019 12:52:50.629 Description: This is a resubmission of a case that was underwritten using the
11/21/2019 12:52:50.629 UW_10.30 KB engine

11/21/2019 12:52:50.629

11/21/2019 12:52:50.629 UWROUTER service will be used for underwriting
11/21/2019 12:52:50.629 ----------------------------------------------------------------
11/21/2019 12:52:50.629 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:50.629 Fault type: Application Severity: Info
11/21/2019 12:52:50.629 Description: This case will be underwritten using UWROUTER 1.0

11/21/2019 12:52:50.629

11/21/2019 12:52:50.629 Calling Checklist1003
11/21/2019 12:52:50.670 ----------------------------------------------------------------
11/21/2019 12:52:50.670 Message type: Code: -1 dec, ffffffff hex
11/21/2019 12:52:50.670 Fault type: Unknown Severity: Info
11/21/2019 12:52:50.670 Description: Begin file type 104 checklist.

11/21/2019 12:52:50.670

11/21/2019 12:52:50.676 ----------------------------------------------------------------
11/21/2019 12:52:50.676 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:50.676 Fault type: Application Severity: Info
11/21/2019 12:52:50.676 Description: Checklist1003 completed successfully

11/21/2019 12:52:50.676

11/21/2019 12:52:50.676 SBAWF13Service_Impl.Underwrite: _creditReptMISMO = '1'
11/21/2019 12:52:50.676 SBAWF13Service_Impl.Underwrite: _MISMOVersion = '2.1'
11/21/2019 12:52:50.684 Loan Type: [01]
11/21/2019 12:52:50.708 ----------------------------------------------------------------
11/21/2019 12:52:50.708 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:50.708 Fault type: Application Severity: Info
11/21/2019 12:52:50.708 Description: createCrReportRequest completed successfully

11/21/2019 12:52:50.708

11/21/2019 12:52:50.708 No Credit Report requests are needed
11/21/2019 12:52:50.708 Calling ChecklistCred
11/21/2019 12:52:50.719 SINGLEIN=0
11/21/2019 12:52:50.732 ----------------------------------------------------------------
11/21/2019 12:52:50.732 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:50.732 Fault type: Application Severity: Info
11/21/2019 12:52:50.732 Description: ChecklistCred completed successfully

11/21/2019 12:52:50.732

11/21/2019 12:52:50.732 Autocopy not requested
11/21/2019 12:52:50.732 command : UNDERWRITE
11/21/2019 12:52:50.732 du underwrite : 1
11/21/2019 12:52:50.732 set BASE_GUID for DUKB service: abbababsbasasas
11/21/2019 12:52:50.732 Set processUndewrriteVars.conversionTarget = RES,HTML
11/21/2019 12:52:50.732 Call IDGen for Doc_Type[1], ID_Type[4], ID_Qty[1]
11/21/2019 12:52:50.749 IDGen returns condition [SUCCESS]
11/21/2019 12:52:50.749 Successfully retrieved new FindingID: 2085534376
11/21/2019 12:52:50.749 command : Underwrite
11/21/2019 12:52:50.749 du underwrite : 1
11/21/2019 12:52:50.749 Performing DU underwrite ...
11/21/2019 12:52:50.749 Calling UWROUTER 1.0 to underwrite the case
11/21/2019 12:52:50.750 Command[KB_Underwrite] set for Underwrite!
11/21/2019 12:52:52.901 Validating reply from UWROUTER_1_0
11/21/2019 12:52:52.901 ----------------------------------------------------------------
11/21/2019 12:52:52.901 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:52.901 Fault type: Application Severity: Info
11/21/2019 12:52:52.901 Description: UWROUTER completed successfully

11/21/2019 12:52:52.901

Re: Splunk querying nested log

dmarling — Thu, 21 Nov 2019 18:00:50 GMT

Actually try this instead:

| makeresults count=1
| eval data="Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 09:21:53.297 Fault type: Application Severity: Info
11/21/2019 09:21:53.297 Description: This is a resubmission of a case that was underwritten using the
11/21/2019 09:21:53.297 UW_10.30 KB engine
11/21/2019 09:21:53.297
11/21/2019 09:21:53.297 UWROUTER service will be used for underwriting
11/21/2019 09:21:53.297 ----------------------------------------------------------------
11/21/2019 09:21:53.297 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 09:21:53.297 Fault type: Application Severity: Info
11/21/2019 09:21:53.297 Description: This case will be underwritten using UWROUTER 1.0
11/21/2019 09:21:53.297
11/21/2019 09:21:53.297 Calling Checklist1003
11/21/2019 09:21:53.345 ----------------------------------------------------------------
11/21/2019 09:21:53.345 Message type: Code: 118310 dec, 1ce26 hex
11/21/2019 09:21:53.345 Fault type: Undefined Severity: Undefined
11/21/2019 09:21:53.345 Description: Hired From Date is missing for secondary employment for
11/21/2019 09:21:53.345 applicant .
11/21/2019 09:21:53.345
11/21/2019 09:21:53.358 -----------------------------------------"
| rex field=data mode=sed "s/([\n\r])\d{2}\/\d{2}\/\d{4} \d{2}:\d{2}:\d{2}.\d{3}/\1/g"
| rex field=data max_match=0 "Calling Checklist[^\n]+\n[^\n]+(?<checklist>[^\-]+)"
| rex field=checklist "Description: (?<Description>[^\e]+)"

I create a "checklist" field that grabs everything under calling checklist1003 between the "-" lines and then extracts the description from that. That is a cleaner way to do it I believe.

Re: Splunk querying nested log

shwetamis — Thu, 21 Nov 2019 18:04:37 GMT

I believe that should work let me try.

Re: Splunk querying nested log

dmarling — Thu, 21 Nov 2019 18:07:25 GMT

It didn't work on your most recent example but this will:

| makeresults count=1
| eval data="11/21/2019 12:52:49.929 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:49.929 Fault type: Application Severity: Info
11/21/2019 12:52:49.929 Description: Verification Successful 
11/21/2019 12:52:49.929 
11/21/2019 12:52:49.929 ----------------------------------------------------------------
11/21/2019 12:52:49.929 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:49.929 Fault type: Application Severity: Info
11/21/2019 12:52:49.929 Description: No errors were detected 
11/21/2019 12:52:49.929 
11/21/2019 12:52:49.929 Performing a CaseInit
11/21/2019 12:52:49.929 LOAN_APPLICATION found. Send to store as Original Mismo 1003!
11/21/2019 12:52:50.604 Validating reply from CaseInit
11/21/2019 12:52:50.604 ----------------------------------------------------------------
11/21/2019 12:52:50.604 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:50.604 Fault type: Application Severity: Info
11/21/2019 12:52:50.604 Description: CaseInit completed successfully 
11/21/2019 12:52:50.604 
11/21/2019 12:52:50.604 Inst ID set for EH-020 [12121231]
11/21/2019 12:52:50.604 Casefile ID [xxxxxxx]
11/21/2019 12:52:50.604 lenderCaseNo [yyyyyyy]
11/21/2019 12:52:50.604 OperStatus_in_mp_casefile_set is not defined.
11/21/2019 12:52:50.604 Calling Credential service to get traits 
11/21/2019 12:52:50.624 ----------------------------------------------------------------
11/21/2019 12:52:50.624 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:50.624 Fault type: Application Severity: Info
11/21/2019 12:52:50.624 Description: InstTraitRequest completed successfully 
11/21/2019 12:52:50.624 
11/21/2019 12:52:50.625 Call RESTBridge for conventional loan with GUID [abbababsbasasas], 
11/21/2019 12:52:50.625 INST_INFO: 12121231|somecompany
11/21/2019 12:52:50.625 
11/21/2019 12:52:50.625 Begin RESTBridgeSubsystem::callRESTBridge() ...
11/21/2019 12:52:50.625 BASE_GUID for VDVS [abbababsbasasas]
11/21/2019 12:52:50.625 Begin RESTBridgeSubsystem::sendRESTBridgeClientRequest...
11/21/2019 12:52:50.625 Call RESTBridgeProxy.sendRESTBridgeRequest: 
11/21/2019 12:52:50.625 Svcname= RESTBridge
11/21/2019 12:52:50.625 Major= 1
11/21/2019 12:52:50.625 Minor= 0
11/21/2019 12:52:50.625 Command= RESTBridge
11/21/2019 12:52:50.628 End RESTBridgeSubsystem::sendRESTBridgeClientRequest
11/21/2019 12:52:50.628 End RESTBridgeSubsystem::callRESTBridge()
11/21/2019 12:52:50.628 SBAWF13Service_Impl:Underwrite:BESTFIT_PRODUCT_SPECIFICATIONS file attachment is not found. This is NOT a multi-product case
11/21/2019 12:52:50.629 Determine ProdAlt case: 
11/21/2019 12:52:50.629 ProdAlt Flag [OFF]
11/21/2019 12:52:50.629 LoanType [01]
11/21/2019 12:52:50.629 prodAltTrait [0]
11/21/2019 12:52:50.629 This is NOT a Product alternative case!
11/21/2019 12:52:50.629 CONVERSION_TARGET: RES,HTML
11/21/2019 12:52:50.629 User format Request: XML,HTML
11/21/2019 12:52:50.629 Determining the underwriting service to be used
11/21/2019 12:52:50.629 Set to use Default engine UWROUTER,1,0 for underwriting
11/21/2019 12:52:50.629 Underwrite::determineUWService::multiProductCase =0
11/21/2019 12:52:50.629 This is not a multi-product case.
11/21/2019 12:52:50.629 ----------------------------------------------------------------
11/21/2019 12:52:50.629 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:50.629 Fault type: Application Severity: Info
11/21/2019 12:52:50.629 Description: This is a resubmission of a case that was underwritten using the
11/21/2019 12:52:50.629 UW_10.30 KB engine 
11/21/2019 12:52:50.629 
11/21/2019 12:52:50.629 UWROUTER service will be used for underwriting
11/21/2019 12:52:50.629 ----------------------------------------------------------------
11/21/2019 12:52:50.629 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:50.629 Fault type: Application Severity: Info
11/21/2019 12:52:50.629 Description: This case will be underwritten using UWROUTER 1.0 
11/21/2019 12:52:50.629 
11/21/2019 12:52:50.629 Calling Checklist1003
11/21/2019 12:52:50.670 ----------------------------------------------------------------
11/21/2019 12:52:50.670 Message type: Code: -1 dec, ffffffff hex
11/21/2019 12:52:50.670 Fault type: Unknown Severity: Info
11/21/2019 12:52:50.670 Description: Begin file type 104 checklist. 
11/21/2019 12:52:50.670 
11/21/2019 12:52:50.676 ----------------------------------------------------------------
11/21/2019 12:52:50.676 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:50.676 Fault type: Application Severity: Info
11/21/2019 12:52:50.676 Description: Checklist1003 completed successfully 
11/21/2019 12:52:50.676 
11/21/2019 12:52:50.676 SBAWF13Service_Impl.Underwrite: _creditReptMISMO = '1'
11/21/2019 12:52:50.676 SBAWF13Service_Impl.Underwrite: _MISMOVersion = '2.1'
11/21/2019 12:52:50.684 Loan Type: [01]
11/21/2019 12:52:50.708 ----------------------------------------------------------------
11/21/2019 12:52:50.708 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:50.708 Fault type: Application Severity: Info
11/21/2019 12:52:50.708 Description: createCrReportRequest completed successfully 
11/21/2019 12:52:50.708 
11/21/2019 12:52:50.708 No Credit Report requests are needed
11/21/2019 12:52:50.708 Calling ChecklistCred
11/21/2019 12:52:50.719 SINGLEIN=0
11/21/2019 12:52:50.732 ----------------------------------------------------------------
11/21/2019 12:52:50.732 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:50.732 Fault type: Application Severity: Info
11/21/2019 12:52:50.732 Description: ChecklistCred completed successfully 
11/21/2019 12:52:50.732 
11/21/2019 12:52:50.732 Autocopy not requested
11/21/2019 12:52:50.732 command : UNDERWRITE
11/21/2019 12:52:50.732 du underwrite : 1
11/21/2019 12:52:50.732 set BASE_GUID for DUKB service: abbababsbasasas
11/21/2019 12:52:50.732 Set processUndewrriteVars.conversionTarget = RES,HTML
11/21/2019 12:52:50.732 Call IDGen for Doc_Type[1], ID_Type[4], ID_Qty[1]
11/21/2019 12:52:50.749 IDGen returns condition [SUCCESS]
11/21/2019 12:52:50.749 Successfully retrieved new FindingID: 2085534376
11/21/2019 12:52:50.749 command : Underwrite
11/21/2019 12:52:50.749 du underwrite : 1
11/21/2019 12:52:50.749 Performing DU underwrite ... 
11/21/2019 12:52:50.749 Calling UWROUTER 1.0 to underwrite the case
11/21/2019 12:52:50.750 Command[KB_Underwrite] set for Underwrite!
11/21/2019 12:52:52.901 Validating reply from UWROUTER_1_0
11/21/2019 12:52:52.901 ----------------------------------------------------------------
11/21/2019 12:52:52.901 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
11/21/2019 12:52:52.901 Fault type: Application Severity: Info
11/21/2019 12:52:52.901 Description: UWROUTER completed successfully 
11/21/2019 12:52:52.901 
"
| rex field=data mode=sed "s/([\n\r])\d{2}\/\d{2}\/\d{4} \d{2}:\d{2}:\d{2}.\d{3}/\1/g"
| rex field=data max_match=0 "Calling Checklist[^\-]+[^\n]+\n[^\n]+(?<checklist>[^\-]+)"
| rex field=checklist "Description: (?<Description>[^\e]+)"

Re: Splunk querying nested log

shwetamis — Wed, 30 Sep 2020 03:03:23 GMT

My Query: index=du sourcetype="du:sbaservice-log" du_service="sbawf1.3" | rex field=data mode=sed "s/([\n\r])\d{2}\/\d{2}\/\d{4} \d{2}:\d{2}:\d{2}.\d{3}/\1/g" | rex field=data max_match=0 "Calling Checklist1003[^\n]+\n[^\n]+(?[^-]+)" | rex field=checklist "Description: (?[^\e]+)" | rex field=_raw "INST_INFO:\s\d+|(?.*)|" | rex field=_raw "lenderCaseNo\s[(?\d+)]" | eval BTime = strptime(Begin_time, "%H:%M:%S.%3N") | eval CTime = strptime(Completion_time, "%H:%M:%S.%3N") | eval ResTime=CTime-BTime

Result:
Description
No validation errors were found

Expected:
Description: Hired From Date is missing for secondary employment for

Re: Splunk querying nested log

dmarling — Thu, 21 Nov 2019 18:47:53 GMT

You still had field=data which was just for my example. This will fix that:

index=du sourcetype="du:sbaservice-log" du_service="sbawf1.3" 
| rex field=_raw mode=sed "s/([\n\r])\d{2}\/\d{2}\/\d{4} \d{2}:\d{2}:\d{2}.\d{3}/\1/g" 
| rex field=_raw max_match=0 "Calling Checklist[^\-]+[^\n]+\n[^\n]+(?<checklist>[^\-]+)" 
| rex field=checklist "Description: (?<Description>[^\e]+)" 
| rex field=_raw "INST_INFO\:\s\d+\|(?.*)\|" 
| rex field=_raw "lenderCaseNo\s[(?\d+)]" 
| eval BTime = strptime(Begin_time, "%H:%M:%S.%3N") 
| eval CTime = strptime(Completion_time, "%H:%M:%S.%3N") 
| eval ResTime=CTime-BTime

Re: Splunk querying nested log

shwetamis — Thu, 21 Nov 2019 18:50:03 GMT

getting error: rror in 'rex' command: Encountered the following error while compiling the regex 'Calling Checklist1003[^\n]+\n[^\n]+(?[^-]+)': Regex: unrecognized character after (? or (?-.

Re: Splunk querying nested log

shwetamis — Wed, 30 Sep 2020 03:03:29 GMT

I corrected it, still not getting expected result

index=du sourcetype="du:sbaservice-log"
| rex field=_raw mode=sed "s/([\n\r])\d{2}\/\d{2}\/\d{4} \d{2}:\d{2}:\d{2}.\d{3}/\1/g"
| rex field=_raw max_match=0 "Calling Checklist1003[^\n]+\n[^\n]+(?[^-]+)"
| rex field=checklist "Description: (?[^\e]+)"
| rex field=_raw "INST_INFO:\s\d+|(?.*)|"
| rex field=_raw "lenderCaseNo\s[(?\d+)]"
| eval BTime = strptime(Begin_time, "%H:%M:%S.%3N")
| eval CTime = strptime(Completion_time, "%H:%M:%S.%3N")
| eval ResTime=CTime-BTime
|table Description lenderInstName lenderCaseNumber Begin_time Completion_time

Re: Splunk querying nested log

shwetamis — Thu, 21 Nov 2019 19:10:57 GMT

seems like its picking the first description from the log line

signal alert flag is 0
Validating Underwrite request

Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
Fault type: Application Severity: Info
Description: No validation errors were found

Re: Splunk querying nested log

dmarling — Thu, 21 Nov 2019 19:21:13 GMT

Are you only wanting stuff after Checklist1003? Running the below run anywhere example returns results I expect:

| makeresults count=1 
| eval _raw="11/21/2019 12:52:49.929 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
 11/21/2019 12:52:49.929 Fault type: Application Severity: Info
 11/21/2019 12:52:49.929 Description: Verification Successful 
 11/21/2019 12:52:49.929 
 11/21/2019 12:52:49.929 ----------------------------------------------------------------
 11/21/2019 12:52:49.929 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
 11/21/2019 12:52:49.929 Fault type: Application Severity: Info
 11/21/2019 12:52:49.929 Description: No errors were detected 
 11/21/2019 12:52:49.929 
 11/21/2019 12:52:49.929 Performing a CaseInit
 11/21/2019 12:52:49.929 LOAN_APPLICATION found. Send to store as Original Mismo 1003!
 11/21/2019 12:52:50.604 Validating reply from CaseInit
 11/21/2019 12:52:50.604 ----------------------------------------------------------------
 11/21/2019 12:52:50.604 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
 11/21/2019 12:52:50.604 Fault type: Application Severity: Info
 11/21/2019 12:52:50.604 Description: CaseInit completed successfully 
 11/21/2019 12:52:50.604 
 11/21/2019 12:52:50.604 Inst ID set for EH-020 [12121231]
 11/21/2019 12:52:50.604 Casefile ID [xxxxxxx]
 11/21/2019 12:52:50.604 lenderCaseNo [yyyyyyy]
 11/21/2019 12:52:50.604 OperStatus_in_mp_casefile_set is not defined.
 11/21/2019 12:52:50.604 Calling Credential service to get traits 
 11/21/2019 12:52:50.624 ----------------------------------------------------------------
 11/21/2019 12:52:50.624 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
 11/21/2019 12:52:50.624 Fault type: Application Severity: Info
 11/21/2019 12:52:50.624 Description: InstTraitRequest completed successfully 
 11/21/2019 12:52:50.624 
 11/21/2019 12:52:50.625 Call RESTBridge for conventional loan with GUID [abbababsbasasas], 
 11/21/2019 12:52:50.625 INST_INFO: 12121231|somecompany
 11/21/2019 12:52:50.625 
 11/21/2019 12:52:50.625 Begin RESTBridgeSubsystem::callRESTBridge() ...
 11/21/2019 12:52:50.625 BASE_GUID for VDVS [abbababsbasasas]
 11/21/2019 12:52:50.625 Begin RESTBridgeSubsystem::sendRESTBridgeClientRequest...
 11/21/2019 12:52:50.625 Call RESTBridgeProxy.sendRESTBridgeRequest: 
 11/21/2019 12:52:50.625 Svcname= RESTBridge
 11/21/2019 12:52:50.625 Major= 1
 11/21/2019 12:52:50.625 Minor= 0
 11/21/2019 12:52:50.625 Command= RESTBridge
 11/21/2019 12:52:50.628 End RESTBridgeSubsystem::sendRESTBridgeClientRequest
 11/21/2019 12:52:50.628 End RESTBridgeSubsystem::callRESTBridge()
 11/21/2019 12:52:50.628 SBAWF13Service_Impl:Underwrite:BESTFIT_PRODUCT_SPECIFICATIONS file attachment is not found. This is NOT a multi-product case
 11/21/2019 12:52:50.629 Determine ProdAlt case: 
 11/21/2019 12:52:50.629 ProdAlt Flag [OFF]
 11/21/2019 12:52:50.629 LoanType [01]
 11/21/2019 12:52:50.629 prodAltTrait [0]
 11/21/2019 12:52:50.629 This is NOT a Product alternative case!
 11/21/2019 12:52:50.629 CONVERSION_TARGET: RES,HTML
 11/21/2019 12:52:50.629 User format Request: XML,HTML
 11/21/2019 12:52:50.629 Determining the underwriting service to be used
 11/21/2019 12:52:50.629 Set to use Default engine UWROUTER,1,0 for underwriting
 11/21/2019 12:52:50.629 Underwrite::determineUWService::multiProductCase =0
 11/21/2019 12:52:50.629 This is not a multi-product case.
 11/21/2019 12:52:50.629 ----------------------------------------------------------------
 11/21/2019 12:52:50.629 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
 11/21/2019 12:52:50.629 Fault type: Application Severity: Info
 11/21/2019 12:52:50.629 Description: This is a resubmission of a case that was underwritten using the
 11/21/2019 12:52:50.629 UW_10.30 KB engine 
 11/21/2019 12:52:50.629 
 11/21/2019 12:52:50.629 UWROUTER service will be used for underwriting
 11/21/2019 12:52:50.629 ----------------------------------------------------------------
 11/21/2019 12:52:50.629 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
 11/21/2019 12:52:50.629 Fault type: Application Severity: Info
 11/21/2019 12:52:50.629 Description: This case will be underwritten using UWROUTER 1.0 
 11/21/2019 12:52:50.629 
 11/21/2019 12:52:50.629 Calling Checklist1003
 11/21/2019 12:52:50.670 ----------------------------------------------------------------
 11/21/2019 12:52:50.670 Message type: Code: -1 dec, ffffffff hex
 11/21/2019 12:52:50.670 Fault type: Unknown Severity: Info
 11/21/2019 12:52:50.670 Description: Begin file type 104 checklist. 
 11/21/2019 12:52:50.670 
 11/21/2019 12:52:50.676 ----------------------------------------------------------------
 11/21/2019 12:52:50.676 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
 11/21/2019 12:52:50.676 Fault type: Application Severity: Info
 11/21/2019 12:52:50.676 Description: Checklist1003 completed successfully 
 11/21/2019 12:52:50.676 
 11/21/2019 12:52:50.676 SBAWF13Service_Impl.Underwrite: _creditReptMISMO = '1'
 11/21/2019 12:52:50.676 SBAWF13Service_Impl.Underwrite: _MISMOVersion = '2.1'
 11/21/2019 12:52:50.684 Loan Type: [01]
 11/21/2019 12:52:50.708 ----------------------------------------------------------------
 11/21/2019 12:52:50.708 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
 11/21/2019 12:52:50.708 Fault type: Application Severity: Info
 11/21/2019 12:52:50.708 Description: createCrReportRequest completed successfully 
 11/21/2019 12:52:50.708 
 11/21/2019 12:52:50.708 No Credit Report requests are needed
 11/21/2019 12:52:50.708 Calling ChecklistCred
 11/21/2019 12:52:50.719 SINGLEIN=0
 11/21/2019 12:52:50.732 ----------------------------------------------------------------
 11/21/2019 12:52:50.732 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
 11/21/2019 12:52:50.732 Fault type: Application Severity: Info
 11/21/2019 12:52:50.732 Description: ChecklistCred completed successfully 
 11/21/2019 12:52:50.732 
 11/21/2019 12:52:50.732 Autocopy not requested
 11/21/2019 12:52:50.732 command : UNDERWRITE
 11/21/2019 12:52:50.732 du underwrite : 1
 11/21/2019 12:52:50.732 set BASE_GUID for DUKB service: abbababsbasasas
 11/21/2019 12:52:50.732 Set processUndewrriteVars.conversionTarget = RES,HTML
 11/21/2019 12:52:50.732 Call IDGen for Doc_Type[1], ID_Type[4], ID_Qty[1]
 11/21/2019 12:52:50.749 IDGen returns condition [SUCCESS]
 11/21/2019 12:52:50.749 Successfully retrieved new FindingID: 2085534376
 11/21/2019 12:52:50.749 command : Underwrite
 11/21/2019 12:52:50.749 du underwrite : 1
 11/21/2019 12:52:50.749 Performing DU underwrite ... 
 11/21/2019 12:52:50.749 Calling UWROUTER 1.0 to underwrite the case
 11/21/2019 12:52:50.750 Command[KB_Underwrite] set for Underwrite!
 11/21/2019 12:52:52.901 Validating reply from UWROUTER_1_0
 11/21/2019 12:52:52.901 ----------------------------------------------------------------
 11/21/2019 12:52:52.901 Message type: SBAWF13Info Code: 1001 dec, 3e9 hex
 11/21/2019 12:52:52.901 Fault type: Application Severity: Info
 11/21/2019 12:52:52.901 Description: UWROUTER completed successfully 
 11/21/2019 12:52:52.901 
 " 
| rex field=_raw mode=sed "s/([\n\r\s]+)\d{2}\/\d{2}\/\d{4} \d{2}:\d{2}:\d{2}.\d{3}/\1/g" 
| rex field=_raw max_match=0 "Calling Checklist[^\-]+[^\n]+\n[^\n]+(?<checklist>[^\-]+)" 
| rex field=checklist "Description: (?<Description>[^\e]+)"

Re: Splunk querying nested log

shwetamis — Thu, 21 Nov 2019 19:28:27 GMT

I only want the description from Calling Checklist1003

Re: Splunk querying nested log

dmarling — Thu, 21 Nov 2019 19:44:28 GMT

I see what happened. This should fix it:

index=du sourcetype="du:sbaservice-log"
| rex field=_raw mode=sed "s/([\n\r\s]+)\d{2}\/\d{2}\/\d{4} \d{2}:\d{2}:\d{2}.\d{3}/\1/g" 
| rex field=_raw max_match=0 "Calling Checklist1003[^\-]+[^\n]+\n[^\n]+(?<checklist>[^\-]+)" 
| rex field=checklist "Description: (?<Description>[^\e]+)"
| rex field=_raw "INST_INFO\:\s\d+\|(?.*)\|"
| rex field=_raw "lenderCaseNo\s[(?\d+)]"
| eval BTime = strptime(Begin_time, "%H:%M:%S.%3N")
| eval CTime = strptime(Completion_time, "%H:%M:%S.%3N")
| eval ResTime=CTime-BTime
|table Description lenderInstName lenderCaseNumber Begin_time Completion_time

Re: Splunk querying nested log

shwetamis — Thu, 21 Nov 2019 20:08:17 GMT

that worked thank you :), so what was the issue ?

Re: Splunk querying nested log

dmarling — Thu, 21 Nov 2019 20:11:12 GMT

The iteration you were using of the "checklist" extract was pulling data from the first line after "Calling Checklist1003" until "-", but in one of the examples there was another line between which broke that logic. I adjusted it to go to the first line AFTER the line of "-" to start the "checklist" field and that fixed it. Sorry about that.