Splunk Search

Trasaction command displaying endswith field before startwith in search results.

SplunkWorthy
Loves-to-Learn Lots

I am attempting to use the transaction field to identify when a  service fails and then the next time it runs successfully after the fail. I am getting results that are close to what I want, but it is showing the results out of order. I am getting the completed status before the faulted status. Here is the command I am using below:

 

transaction ID startswith="Faulted" endswith="Completed"

 

My goal is to group the logs by ID where it shows me the first faulted log followed by the completed log but its giving me the results in reverse order. I have already tried to sort by _time and the status field before the transaction command but that does not help.

Labels (3)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @SplunkWorthy ,

could you share a sample of your resulta ans logs?

Ciao.

Giuseppe

 

0 Karma

SplunkWorthy
Loves-to-Learn Lots

I am unable to provide a copy of the logs, but I can create an example myself:

 

_time                                 Name       ID          status     message         start time                         endtime

2026-05-22  00:00:30:900                    Log1            123                     4                      Complete            2026-05-22T04:00:30.900Z                  2026-05-22T04:00:40.03Z                                                                                                                                               5                    Failed         2026-05-22T08:02:00.888Z               2026-05-22T08:02:10.933Z

In this example, the fail is showing with the completion that happened prior to the failure, when instead, I want it to show the completion after. This example shows the results after they have been tabled.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

In transaction command reference it Is said

Descending chronological order required

The transaction command requires that the incoming events be in descending chronological order. Some commands, such as eval, might change the order or time labeling of events. If one of these commands precedes the transaction command, your search returns an error unless you include a sort command in your search. The sortcommand must occur immediately before the transaction command to reorder the search results in descending chronological order.

So you must sort those into correct order before you can use successfully transaction command.

0 Karma

SplunkWorthy
Loves-to-Learn Lots

These are the results when I use a sort before the transaction command:

 

sort -status    = no results

sort -message   = no results

sort -time  = sorted based on time but no change to logs that are used in the results. Still complete then fail.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

You need to convert start time to epoch and then sort by it.

| eval sTime = strptime(startTime,"%FT%T.%3Q%Z")
0 Karma

SplunkWorthy
Loves-to-Learn Lots

It seems that syntax does not successfully convert StartTime to epoch. When you table the sTime , no results are visible and the sort does not work. I also tired

| eval sTime = strptime(startTime,"%FT%T.%QZ")

and

| eval sTime = strptime(StartTime, "%Y-%m-%dT%H:%M%S.%3Q%Z)

and

| eval sTime = strptime(StartTime, "%Y-%m-%dT%H:%M%S.%QZ)

and

| eval sTime = strftime(StartTime, "%Y-%m-%dT%H:%M%S.%QZ)

0 Karma

bowesmana
SplunkTrust
SplunkTrust

In your earlier post when you said you are using 

| sort -time

if that is exactly what you used then unless you have a field called time (as opposed to Splunk's _time)  then you are not sorting anything.

You may also be better to use

| transaction ID startswith="message=Faulted" endswith="message=Completed"

if message is the field containing what you are basing the transaction on.

Hhowever, transaction is not always the best command to use as it can give wrong or no results, depending on your data size. This is particularly important when not using maxspan to limit what data you are searching and the number of events you have.

Also note that you should use 

| sort 0 _time

otherwise it will truncate data.

 

SplunkWorthy
Loves-to-Learn Lots

Yes, I was referring to Splunk's _time.

 

| transaction ID startswith="message=Faulted" endswith="message=Completed"

This search returns no results.

 

This sort does not change the results.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Either there is something more going on with your data than you're showing or you're sloppy in copy-pasting.

Earlier you showed us supposed contents of your event having a "message" field with a value of "Complete". Now you're searching for a string "Completed" there. If both your event and your search were literal copies from your environment, they have no thance of matching one another.

 

0 Karma

SplunkWorthy
Loves-to-Learn Lots

The latter of the two is an example I created that is very similar to the actual search results I am seeing.

0 Karma

bowesmana
SplunkTrust
SplunkTrust

It's hard to provide a solution without a good level of detail. What fields are available? Can you post a redacted if necessary screen shot of your data, because the detail is important.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

SOC4Kafka - New Kafka Connector Powered by OpenTelemetry

The new SOC4Kafka connector, built on OpenTelemetry, enables the collection of Kafka messages and forwards ...

Rounding off the Splunk Dashboard Contest

What does a contest-winning Splunk dashboard look like? In this case, it isn't in a browser tab at all. It ...

A Four Part Event Series: AI + Observability: AI Agents, LLMs, Apps, & Infrastructure

AI + Observability: AI Agents, LLMs, Apps, & Infrastructure The rapid evolution of artificial intelligence ...