topic OpenTelemetry business transactions only show HTTP Post in Splunk AppDynamics

OpenTelemetry business transactions only show HTTP Post

Björn_Wenzel — Sat, 29 Oct 2022 17:33:23 GMT

For 3 Spring Boot applications (2 classic servlet and one spring cloud gateway in front) I have enabled OTEL and see the first metrics arriving successfully in my Controller.

Due to whatever reasons it seems that the Business Transactions only show that the app is NodeJS & the name of the Transaction is HTTP Post. I don't see the different routes that are available via the gateway & routed to the 2 classic spring boot applications.

For a short test I also reconfigured the collector to log every request instead of forwarding it to the controller, please see attached details about the request:

Resource attributes:
     -> appdynamics.controller.account: Str(acme)
     -> appdynamics.controller.host: Str(acme.saas.appdynamics.com)
     -> appdynamics.controller.port: Int(443)
     -> service.name: Str(gateway)
     -> service.namespace: Str(acme-namespace)
     -> telemetry.sdk.language: Str(java)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.16.0)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope org.springframework.cloud.sleuth
Span #0
    Trace ID       : 5fea943a828ccd2fff3072992d4e5ab6
    Parent ID      :
    ID             : 5a8b847d55a33279
    Name           : HTTP GET
    Kind           : SPAN_KIND_SERVER
    Start time     : 2022-10-29 17:19:45.595667 +0000 UTC
    End time       : 2022-10-29 17:19:45.640038791 +0000 UTC
    Status code    : STATUS_CODE_ERROR
    Status message :
Attributes:
     -> http.host: Str(localhost:8881)
     -> http.method: Str(GET)
     -> http.path: Str(/api/acme/endpoint)
     -> http.route: Str(/api/acme/endpoint)
     -> http.scheme: Str(http)
     -> http.status_code: Int(500)
     -> http.target: Str(/api/acme/endpoint)
     -> http.user_agent: Str(curl/7.79.1)
     -> net.host.name: Str(localhost)
     -> net.transport: Str(http)
Events:
SpanEvent #0
     -> Name: exception
     -> Timestamp: 2022-10-29 17:19:45.637675125 +0000 UTC
     -> DroppedAttributesCount: 0
     -> Attributes::
          -> exception.message: Str(Connection refused: localhost/[0:0:0:0:0:0:0:1]:8884)
          -> exception.stacktrace: Str(io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/[0:0:0:0:0:0:0:1]:8884
	Suppressed: The stacktrace has been enhanced by Reactor, refer to additional information below:
Error has been observed at the following site(s):
	*__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.authentication.AuthenticationWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
Original Stack Trace:
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)
)
          -> exception.type: Str(io.netty.channel.AbstractChannel.AnnotatedConnectException)

Any idea?

Re: OpenTelemetry business transactions only show HTTP Post

Satbir_Singh — Fri, 12 May 2023 17:09:25 GMT

Hi,

It seems that the Business Transactions in the AppDynamics Controller are not able to differentiate between the different routes that are available via the gateway and routed to the 2 classic Spring Boot applications. This is likely because the routes are not being properly instrumented with OTEL.

To fix this issue, you should make sure that all the necessary libraries are being instrumented with OTEL and that the necessary spans are being created for each route. You may also need to check the configuration of the OTEL collector to ensure that it is properly forwarding all the data to the AppDynamics Controller.

Looking at the log you provided, it seems that there was an error connecting to a service running on port 8884. This could be related to the issue you are experiencing with the Business Transactions not being properly identified. You may need to investigate this error further and resolve it before the Business Transactions can be properly instrumented.

Thanks,
Satbir Singh