Traces not showing up in APM with Splunk Observability Trial

[AndrewBriggs]

I am calling the trace endpoint https://ingest.<realm>.signalfx.com/v2/trace/signalfxv1 and sending this span in the body:

[
{
"id": "003cfb6642471ba4",
"traceId": "0025ecb5dc31498b931bce60be0784cd",
"name": "reloadoptionsmanager",
"timestamp": 1716477080494000,
"kind": "SERVER",
"remoteEndpoint": {
"serviceName": "XXXXXXX"
},
"Tags": {
"data": "LogData",
"eventId": "0",
"severity": "8"
}
}
]

The request receives a 200 response. The response body is OK. However the span does not appear in APM.

timestamp is the the number of microseconds since 1/1/1970.

[bishida · Accepted solution]

Manually sending a span using something like curl can often be trickier than instrumenting a sample application. But, that said, it is possible. I see you're using the zipkin compatible endpoint.

This page may help if you want to try other formats/endpoints:
https://docs.splunk.com/observability/en/apm/apm-spans-traces/span-formats.html

Did you replace <realm> with your realm? e.g. "us1" or "us0", etc

With a single span, you won't see anything on the service map, but you should see something on the APM home screen. Here is a sample span that worked for me:

curl -X POST https://ingest.us1.signalfx.com/v2/trace/signalfxv1 \
-H 'Content-Type: application/json' \
-H 'X-SF-TOKEN: XXXXXXXXXXXXXXXXXX' \
-d '[{"traceId": "a03ee8fff1dcd9b9","id": "2e8cfb154b59a41f","kind": "SERVER","name": "post /location/update/v4","timestamp": 1716495917000,"duration": 131848,"localEndpoint": {"serviceName": "routing"},"tags": {"ecosystem": "prod","habitat": "uswest1aprod","http.uri.client": "/location/update/v4","region": "uswest1-prod","response_status_code": "200"},"shared": true}]'