So I have two different services where an API call starts from service A and propagates to service B. I want to trace the errors for this and creating a dashboard to show the consolidated errors. Logs are as follows.  Service A logs: 10.0.9.456 - - 23/Mar/2021:17:29:52 +0000 "POST Error occured in service A status 400 bad request referenceid 1615 msg Some bad request error occured in application B status 400 url /test/user/myuserfield/authorize?service=myservicename&serviceT=myserviceTypeid Service B logs:     { "userId": "/myuserfield", "transactionId": "abcd", "timestamp": "2021-03-24T15:41:25.770Z", "eventName": "myevent", "component": "mycomponent", "response": { "statusCode": "400", "detail": { "reason": "Bad Request" } }, "http": { "request": { "method": "POST", "path": "http://dummyurl", "queryParameters": { "serviceId": [ "myservicename" ], "serviceType": [ "myservicetype" ] } } } }       So the mapping of fields between these two service is as follows: Service A Field Service B Field status statusCode service serviceId serviceType serviceT user userId    I have tried to use subsearch extract fields from service A :   index=*serviceB* | spath | rename userId as user, http.request.queryParameters.serviceId{} as service, http.request.queryParameters.serviceType{} as serviceT | search [search index=*serviceA* | rex "/test/user(?<user>/\w+)+/authorize.*\?+service+\=(?<service>\w+)+\&+serviceT\=(?<serviceT>.*)\"" | dedup user service serviceT | fields user service serviceT] Above expression provides me the logs of Service B which are propagated from service A. What i want now is to display the data in Tabular format for better readability. So i have two questions: 1. Am i going with the right approach by using subsearch here ? Is the expression seems to be correct and best possible solution? 2. Above expression provides me different error logs for different users in json format. How do i convert these to tabular format having userId, time, status etc. ? I also want to filter my table based on multiple status filters (like 4XX, 5XX etc)..how to achieve that? ... View more

I have 4 applications integrated with each other -  their names let's say A, B, C, D respectively. All these applications have different index, sourcetype and source. Now, when i call my API from application A, it traverse to B, C and D to perform some operations. I want to trace the error logs of all these applications at one location in splunk. Also, if an error occurs, there is a common reference id which is logged in all the application's logs respectively. How to achieve this ? Is there a straightforward concept for this type of scenario? What i am imagining is like this : Ref ID Application A Logs Application B  Logs Application C  Logs Application D  Logs 1234 <logs of app A for id 1234> <logs of app B for id 1234> <logs of app C for id 1234> <logs of app D for id 1234> 4567 <logs of app A for id 4567> <logs of app B for id 4567> <logs of app C for id 4567> <logs of app D for id 4567>     Or is there a better way of doing this? I am trying to create a dashboard panel with this result. ... View more