Splunk Search

How can I display a chain of all ten messages?

SajarKumarPat
New Member

I have a message thread, these messages are coming on splunk.
The chain consists of ten different messages: five messages from one system, five messages from another (backup) system.

Messages from the primary system use the same SrcMsgId value, and messages from the backup system are combined with a common SrcMsgId.
Messages from the standby system also have a Mainsys_srcMsgId value - this value is identical to the main system's SrcMsgId value.

The message chain from the backup system enters the splunk immediately after the messages from the main system.
Tell me how can I display a chain of all ten messages? Perhaps first messages from the first system (main), then from the second (backup) with the display of the time of arrival at the server.

With time, I understand, I will include _time in the request. I got a little familiar with the syntax of queries, but still I still have a lot of difficulties with creating queries.
Please help me with an example of the correct request.
Thank you in advance!

Labels (1)
Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @SajarKumarPat,

if you want all the content of all tem messages, you have to use the transaction command but it's very slow:

index=your_index source=primary_system
| transaction SrcMsgId
| append [ search  
   index=your_index source=backup_system
   | transaction SrcMsgId
   | rename SrcMsgId AS backup_SrcMsgId
   | rename Mainsys_srcMsgId AS SrcMsgId ]
| transaction SrcMsgId

It's possible another more performant solution using stats but it's more complicated and must be tested:

index=your_index (source=primary_system OR  source=backup_system)
| eval 
   backup_SrcMsgId=if(source=backup_system,SrcMsgId,""),
   SrcMsgId=if(source=primary_system,SrcMsgId,Mainsys_srcMsgId)
| stats values(_raw) AS _raw BY SrcMsgId

Ciao.

Giuseppe

0 Karma

metylkinandrey
Communicator

We decided to change the key: "sourcetype" of the backup system to a unique one.
Let the main one have the following value: source="testsystem-2"
For the backup: source="testsystem-3"
I tried the following queries again, now I understand their logic:

index="bl_logging" source="testsystem-2"
| transaction SrcMsgId
| append [ search
     index="bl_logging" source="testsystem-2"
     | transaction SrcMsgId
     | rename SrcMsgId AS backup_SrcMsgId
     | rename Mainsys_srcMsgId AS SrcMsgId ]
| transaction SrcMsgId

Result: No results found.

index="bl_logging" source="testsystem-2"
| eval
     backup_SrcMsgId=if(source="testsystem-3",SrcMsgId,""),
     SrcMsgId=if(source="testsystem-2",SrcMsgId,Mainsys_srcMsgId)
| stats values(_raw) AS _raw BY SrcMsgId

Result: No results found.

I have verified that messages from both systems are written to the same index.

 

 

0 Karma

metylkinandrey
Communicator

@gcusello Thank you very much for the answer!

We tried like this:

```

index="bl_logging" sourcetype="testsystem-2"
| table _time srcMsgId Mainsys_srcMsgId messageId
| transaction SrcMsgId
| append [ search
| transaction SrcMsgId
| rename SrcMsgId AS backup_SrcMsgId
| rename Mainsys_srcMsgId AS SrcMsgId ]
| transaction SrcMsgId
| eval
backup_SrcMsgId=if(source=testsystem-2,SrcMsgId,""),
SrcMsgId=if(source=testsystem-2,SrcMsgId,Mainsys_srcMsgId)
| stats values(_raw) AS _raw BY SrcMsgId

```

We look at the message from both systems (main and backup) in the source sourcetype="testsystem-2"
Also, for convenient visualization, we add: | table _time srcMsgId Mainsys_srcMsgId messageId.

As a result, we get: No results found.
Tell me please what are we doing wrong?

We wanted to see all ten messages one after the other, in the order in which they arrived at the server. Five from the main system, for example, combined "srcMsgId": "rwfsdfsfqwe121432gsgsfgd71" and five from the backup: "srcMsgId": "rwfsdfsfqwe121432gsgsfgd72". The problem is that messages from other systems also come to the server, all messages are mixed (chaotically), which is why we want to organize all messages from one system and its relative in the application. Messages from the backup system are associated with the main system only by this parameter: "Mainsys_srcMsgId": "rwfsdfsfqwe121432gsgsfgd71" - using this key, we understand that messages come from the backup system (secondary to the main one).
Examples of messages from the primary and secondary system:

Main system:

```

{
"event": "Sourcetype test please",
"sourcetype": "testsystem-2",
"host": "some-host-123",
"fields":
{
"messageId": "ED280816-E404-444A-A2D9-FFD2D171F32",
"srcMsgId": "rwfsdfsfqwe121432gsgsfgd71",
"Mainsys_srcMsgId": "",
"baseSystemId": "abc1",
"routeInstanceId": "abc2",
"routepointID": "abc3",
"eventTime": "1985-04-12T23:20:50Z",
"messageType": "abc4",
"GISGMPRequestID": "PS000BA780816-E404-444A-A2D9-FFD2D1712345",
"GISGMPResponseID": "PS000BA780816-E404-444B-A2D9-FFD2D1712345",
"resultcode": "abc7",
"resultdesc": "abc8"
}
}

```

Message from backup system:

```

{
"event": "Sourcetype test please",
"sourcetype": "testsystem-2",
"host": "some-host-123",
"fields":
{
"messageId": "ED280816-E404-444A-A2D9-FFD2D171F23",
"srcMsgId": "rwfsdfsfqwe121432gsgsfgd72",
"Mainsys_srcMsgId": "rwfsdfsfqwe121432gsgsfgd71",
"baseSystemId": "abc1",
"routeInstanceId": "abc2",
"routepointID": "abc3",
"eventTime": "1985-04-12T23:20:50Z",
"messageType": "abc4",
"GISGMPRequestID": "PS000BA780816-E404-444A-A2D9-FFD2D1712345",
"GISGMPResponseID": "PS000BA780816-E404-444B-A2D9-FFD2D1712345",
"resultcode": "abc7",
"resultdesc": "abc8"
}
}

```

When we want to combine in a request only five messages from one chain combined: "srcMsgId".
We make a request like this:

```

index="bl_logging" sourcetype="testsystem-2"
| transaction maxpause=5m srcMsgId Mainsys_srcMsgId messageId
| table _time srcMsgId Mainsys_srcMsgId messageId duration eventcount
| sort srcMsgId_time
| streamstats current=f window=1 values(_time) as prevTime by subject
| eval timeDiff=_time-prevTime
| delta _time as timediff

```

It works great, but right now we need to see a thread of ten messages.

 

Thank you in advance!
Andrew.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...