Java serverless not correlating due to due to potentially malformed header

[Catherine_Doce]

Hi,

I'm working on instrumenting Java Lambda. My environment contains 2 lambdas. the first one is triggering the second one through SNS.

Also, implemented the sns exit call from the first lambda.

Now, I'm trying to correlate the business transaction to be the same as started from the first lambda and passing to the second one.

I'm using Manual Instrumentation and followed AppDynamics documentation:

https://docs.appdynamics.com/display/PRO45/Manual+Tracer+Instrumentation

This is a snippet from my code on how I'm fetching the correlation header from an input:

public Context handleRequest(InputStream input, Context context) {

    if (runAppDynamics) {


        //Instantiate the tracer
        Tracer tracer = AppDynamics.getTracer(context);

        //Automatically Parse the correlation header
        InputStream inputStream = InputStreamConverter.convertToMarkSupportedInputStream(input);


        try {
            String inputAsString = IOUtils.toString(inputStream, Charset.forName("UTF-8"));
            com.appdynamics.serverless.tracers.dependencies.com.google.gson.JsonParser parser = new JsonParser();
            JsonObject inputObject = parser.parse(inputAsString).getAsJsonObject();

            if (inputObject.has(Tracer.APPDYNAMICS_TRANSACTION_CORRELATION_HEADER_KEY)) {
                System.out.println("corrHeader in tracer");
                correlationHeader = inputObject.get(Tracer.APPDYNAMICS_TRANSACTION_CORRELATION_HEADER_KEY).getAsString();
            } else {
                //Try reading from HTTP headers
                if (inputObject.has("headers")) {
                    System.out.println("corrHeader in headers");
                    JsonObject httpHeaders = inputObject.getAsJsonObject("headers");
                    if (httpHeaders.has(Tracer.APPDYNAMICS_TRANSACTION_CORRELATION_HEADER_KEY)) {
                        correlationHeader = httpHeaders.get(Tracer.APPDYNAMICS_TRANSACTION_CORRELATION_HEADER_KEY).getAsString();
                    }
                }
            }
        }
        catch (IOException e){}

        if(correlationHeader == null) {
            correlationHeader = "corrheadertest";
        }
        //Create Transaction
        transaction = tracer.createTransaction(correlationHeader);

        //Start the transaction monitoring.
        transaction.start();

 This is the Lambda Test input:

{
"key": "Content-Type",
"singularityheader": "com.appdynamics.serverless.tracers.aws.correlation.CorrelationHeader"
}

Every time I test my Lambda it didn't correlate and cloud watch logs return:

[AppDynamics Tracer] [DEBUG]: Creating transaction for header [com.appdynamics.serverless.tracers.aws.correlation.CorrelationHeader]

[AppDynamics Tracer] [DEBUG]: Processing continuing transaction => com.appdynamics.serverless.tracers.aws.correlation.CorrelationHeader

[AppDynamics Tracer] [DEBUG]: Caller chains read => []

[AppDynamics Tracer] [DEBUG]: Not correlating transaction due to potentially malformed header [com.appdynamics.serverless.tracers.aws.correlation.CorrelationHeader]

I tried to change the value many times to represent valid correlation header, but it gives me the same message for every value I put.

Any idea what I'm missing here and what should be a valid correlation header?

Thanks

[Peter_Holditch]

Catherine,

It appears that the value of the correlation header you are trying to consume on input is "com.appdynamics.serverless.tracers.aws.correlation.CorrelationHeader"

That is definitely not a valid header, where did you obtain it from?

To give you an idea of what you should be expecting to see, an example of a valid header is:

appId=4113*ctrlguid=1539290818*acctguid=e2c8a068-be66-46a4-a0d3-aeff5f18ecc9*btid=532007*guid=efbe5515-1e93-496c-9f53-47c1640069e73*ts=34869499416608*cidfrom=7500*unresolvedexitid=139411*exitguid=40*cidto={[UNRESOLVED][139411]}*etypeorder=HTTP

You should get a string of that form returned to you when you call getCorrelationHeader() on the ExitCall object you created in the upstream lambda.

Warm regards,

Peter

[Catherine_Doce]

Hi Peter,

I passed a correct formate header to the first lambda and it's succeeded to pass it to the second lambda and correlate the transaction created from the first lambda, but I found that, the first Lambda is correlating to itself.

I want the first Lambda to start the Transaction and generate the correlation header, then fetch this header and send it to the second Lambda.

Any idea how I can fetch the generated header from first Lambda? what Package in AppD I can call it to get the generated header?

Thanks

[Peter_Holditch · Accepted solution]

Catherine,

You should create an exit call object and obtain the outgoing correlation header from that, as shown in the code sample in the documentation here: https://docs.appdynamics.com/display/PRO45/.Manual+Tracer+Instrumentation+v4.5.x#id-.ManualTracerIns...

I pasted the relevant section below:

//Define the createExitCall method to obtain an exitCall object.    

ExitCall exitCall = transaction.createExitCall(callType, identifyingProperties);

outgoingHeader =  exitCall.getCorrelationHeader();

exitCall.start();

try {

// Tracer.APPDYNAMICS_TRANSACTION_CORRELATION_HEADER_KEY is the name of the header that should be set

    if (outgoingHeader != null) {

        conn.setRequestProperty(Tracer.APPDYNAMICS_TRANSACTION_CORRELATION_HEADER_KEY, outgoingHeader); // set the correlation header on an HttpURLConnection

    }

// Make the exit call here

} finally {

        exitCall.stop();

    }

}

[Catherine_Doce]

Great Peter, exitcall.getCorrelationHeader(), this what I was looking for.

Noe the first Lambda generates the correlation header then fetch the outgoing header and send it to each Lambda through it's exit call.

Thanks a lot for your help, I'll mark this case as solved.

Please let me know if I need to do anything else, like put ranks for you or anything else should be done when I'm closing a post. This is my first post on AppD community.

Thanks

[Peter_Holditch]

Catherine,

I'm delighted that got you going.  

If you have any suggestions as to how we could make this easier to find in the documentation, please let me know.  From the inside, it's easy to take too much for granted.  What search terms did you try to use to find this?   Where would you expect to see it?

Warm, regards,

Peter

[Catherine_Doce]

Thanks Peter for your reply.

Actually, I don't know how the correlation header should look like. I tried many entries, but in every time it returns malformed header and generate a new one like:

[AppDynamics Tracer] [DEBUG]: Correlation Header generated => com.appdynamics.serverless.tracers.aws.correlation.CorrelationHeader@3fee9989

This is why I used it same as the generated value com.appdynamics.serverless.tracers.aws.correlation.CorrelationHeader without the sequence number at the end.

Yes, I get a header chain generated after that and looks like that:

HeaderChain{root='appId=300*ctrlguid=1570069847*acctguid=a678f643-40d2-4d5d-8a32-636b904e6186*ts=1582757985080*btid=145515*guid=98faae3f-ddfb-4904-bc01-19078418956d', fromCompChain='174213', toCompChain='', exitTypeCallChain='', exitSubtypeCallChain='', snapshotEnabledSet=false, fromCompIds=[174213], threadCallChainForOutOfProcess='null'}

But, the generated correlation header is not fetched by the second Lambda. Second Lambda code:

public String handleRequest(SNSEvent event, Context context) {

        String snsMsg = "Hello From Lambda To SNS";

       String messasge =  event.getRecords().get(0).getSNS().getMessage();

       String correlationHeader="";

        //Instantiate the tracer
        Tracer tracer = AppDynamics.getTracer(context);


        com.appdynamics.serverless.tracers.dependencies.com.google.gson.JsonParser parser = new JsonParser();
        JsonObject inputObject = parser.parse(messasge).getAsJsonObject();

        if (inputObject.has(Tracer.APPDYNAMICS_TRANSACTION_CORRELATION_HEADER_KEY)) {
            System.out.println("corrHeader in tracer");
            correlationHeader = inputObject.get(Tracer.APPDYNAMICS_TRANSACTION_CORRELATION_HEADER_KEY).getAsString();
            System.out.println(correlationHeader);
        } else {
            //Try reading from HTTP headers
            if (inputObject.has("headers")) {
                System.out.println("corrHeader in headers");
                JsonObject httpHeaders = inputObject.getAsJsonObject("headers");
                if (httpHeaders.has(Tracer.APPDYNAMICS_TRANSACTION_CORRELATION_HEADER_KEY)) {
                    correlationHeader = httpHeaders.get(Tracer.APPDYNAMICS_TRANSACTION_CORRELATION_HEADER_KEY).getAsString();
                    System.out.println(correlationHeader);
                }
            }
        }

        //Create Transaction
        Transaction transaction = tracer.createTransaction(correlationHeader);

        //Start the transaction monitoring.
        transaction.start();

        //Lambda code
        try{
            context.getLogger().log("SNS Message: " + snsMsg);
            context.getLogger().log(event.getRecords().get(0).getSNS().getMessage());
            return null;
        }

        finally {
            transaction.stop();
            AppDynamics.cleanup();
        }
    }
}

Cloud watch logs that showing receiving message to this Lambda:

{
    "singularityheader": "com.appdynamics.serverless.tracers.aws.correlation.CorrelationHeader",
    "msg": "Hello from InitSample to SNS"
}

I'm still missing something on how to pass the generated header from the first lambda to the second one?

Thanks