How can I display a chain of all ten messages?

[SajarKumarPat]

I have a message thread, these messages are coming on splunk.
The chain consists of ten different messages: five messages from one system, five messages from another (backup) system.

Messages from the primary system use the same SrcMsgId value, and messages from the backup system are combined with a common SrcMsgId.
Messages from the standby system also have a Mainsys_srcMsgId value - this value is identical to the main system's SrcMsgId value.

The message chain from the backup system enters the splunk immediately after the messages from the main system.
Tell me how can I display a chain of all ten messages? Perhaps first messages from the first system (main), then from the second (backup) with the display of the time of arrival at the server.

With time, I understand, I will include _time in the request. I got a little familiar with the syntax of queries, but still I still have a lot of difficulties with creating queries.
Please help me with an example of the correct request.
Thank you in advance!

[gcusello]

Hi @SajarKumarPat,

if you want all the content of all tem messages, you have to use the transaction command but it's very slow:

index=your_index source=primary_system
| transaction SrcMsgId
| append [ search  
   index=your_index source=backup_system
   | transaction SrcMsgId
   | rename SrcMsgId AS backup_SrcMsgId
   | rename Mainsys_srcMsgId AS SrcMsgId ]
| transaction SrcMsgId

It's possible another more performant solution using stats but it's more complicated and must be tested:

index=your_index (source=primary_system OR  source=backup_system)
| eval 
   backup_SrcMsgId=if(source=backup_system,SrcMsgId,""),
   SrcMsgId=if(source=primary_system,SrcMsgId,Mainsys_srcMsgId)
| stats values(_raw) AS _raw BY SrcMsgId

Ciao.

Giuseppe

[metylkinandrey]

We decided to change the key: "sourcetype" of the backup system to a unique one.
Let the main one have the following value: source="testsystem-2"
For the backup: source="testsystem-3"
I tried the following queries again, now I understand their logic:

index="bl_logging" source="testsystem-2"
| transaction SrcMsgId
| append [ search
     index="bl_logging" source="testsystem-2"
     | transaction SrcMsgId
     | rename SrcMsgId AS backup_SrcMsgId
     | rename Mainsys_srcMsgId AS SrcMsgId ]
| transaction SrcMsgId

Result: No results found.

index="bl_logging" source="testsystem-2"
| eval
     backup_SrcMsgId=if(source="testsystem-3",SrcMsgId,""),
     SrcMsgId=if(source="testsystem-2",SrcMsgId,Mainsys_srcMsgId)
| stats values(_raw) AS _raw BY SrcMsgId

Result: No results found.

I have verified that messages from both systems are written to the same index.

[metylkinandrey]

@gcusello Thank you very much for the answer!

We tried like this:

```

index="bl_logging" sourcetype="testsystem-2"
| table _time srcMsgId Mainsys_srcMsgId messageId
| transaction SrcMsgId
| append [ search
| transaction SrcMsgId
| rename SrcMsgId AS backup_SrcMsgId
| rename Mainsys_srcMsgId AS SrcMsgId ]
| transaction SrcMsgId
| eval
backup_SrcMsgId=if(source=testsystem-2,SrcMsgId,""),
SrcMsgId=if(source=testsystem-2,SrcMsgId,Mainsys_srcMsgId)
| stats values(_raw) AS _raw BY SrcMsgId

```

We look at the message from both systems (main and backup) in the source sourcetype="testsystem-2"
Also, for convenient visualization, we add: | table _time srcMsgId Mainsys_srcMsgId messageId.

As a result, we get: No results found.
Tell me please what are we doing wrong?

We wanted to see all ten messages one after the other, in the order in which they arrived at the server. Five from the main system, for example, combined "srcMsgId": "rwfsdfsfqwe121432gsgsfgd71" and five from the backup: "srcMsgId": "rwfsdfsfqwe121432gsgsfgd72". The problem is that messages from other systems also come to the server, all messages are mixed (chaotically), which is why we want to organize all messages from one system and its relative in the application. Messages from the backup system are associated with the main system only by this parameter: "Mainsys_srcMsgId": "rwfsdfsfqwe121432gsgsfgd71" - using this key, we understand that messages come from the backup system (secondary to the main one).
Examples of messages from the primary and secondary system:

Main system:

```

{
"event": "Sourcetype test please",
"sourcetype": "testsystem-2",
"host": "some-host-123",
"fields":
{
"messageId": "ED280816-E404-444A-A2D9-FFD2D171F32",
"srcMsgId": "rwfsdfsfqwe121432gsgsfgd71",
"Mainsys_srcMsgId": "",
"baseSystemId": "abc1",
"routeInstanceId": "abc2",
"routepointID": "abc3",
"eventTime": "1985-04-12T23:20:50Z",
"messageType": "abc4",
"GISGMPRequestID": "PS000BA780816-E404-444A-A2D9-FFD2D1712345",
"GISGMPResponseID": "PS000BA780816-E404-444B-A2D9-FFD2D1712345",
"resultcode": "abc7",
"resultdesc": "abc8"
}
}

```

Message from backup system:

```

{
"event": "Sourcetype test please",
"sourcetype": "testsystem-2",
"host": "some-host-123",
"fields":
{
"messageId": "ED280816-E404-444A-A2D9-FFD2D171F23",
"srcMsgId": "rwfsdfsfqwe121432gsgsfgd72",
"Mainsys_srcMsgId": "rwfsdfsfqwe121432gsgsfgd71",
"baseSystemId": "abc1",
"routeInstanceId": "abc2",
"routepointID": "abc3",
"eventTime": "1985-04-12T23:20:50Z",
"messageType": "abc4",
"GISGMPRequestID": "PS000BA780816-E404-444A-A2D9-FFD2D1712345",
"GISGMPResponseID": "PS000BA780816-E404-444B-A2D9-FFD2D1712345",
"resultcode": "abc7",
"resultdesc": "abc8"
}
}

```

When we want to combine in a request only five messages from one chain combined: "srcMsgId".
We make a request like this:

```

index="bl_logging" sourcetype="testsystem-2"
| transaction maxpause=5m srcMsgId Mainsys_srcMsgId messageId
| table _time srcMsgId Mainsys_srcMsgId messageId duration eventcount
| sort srcMsgId_time
| streamstats current=f window=1 values(_time) as prevTime by subject
| eval timeDiff=_time-prevTime
| delta _time as timediff

```

It works great, but right now we need to see a thread of ten messages.

Thank you in advance!
Andrew.