Security

How do i track a call across various indexes

sriramfcam
New Member

Each API call has a unique transactionID. The api call goes thru various indexes. How do i track the progress of the call (ie when the call enters and leaves each index)?
The problem is made harder by the fact that sometimes a call goes from index1 to index2, returns to index1, then goes to index 3.....and so on

Tags (1)
0 Karma

skoelpin
SplunkTrust
SplunkTrust

First question, why are you having the data go to different indexes? I'm assuming your segregating the data based off the application its coming from?

Now to solve your problem, you can simply include all 3 indexes in your initial search like this

index=index1 OR index=index2 OR index=index3

This will search all 3 indexes and bring back data which you can than massage into a search to track the transaction

0 Karma

sriramfcam
New Member

Distributed apps - hence the different indexes

That solution does not help out if we have back and forth calls between indexes

I'm doing the following
|stats earliest(_time) as starttime, latest(_time) as stoptime by index |sort starttime

This gives me

index
index1 1545851255 1545851259

index2 1545851256 1545851257

index3 1545851257 1545851258

This does not show each entry and exit into indexes...
I would like the output to be something like

index1 1545851255 1545851256

index2 1545851256 1545851257

index3 1545851257 1545851258
index1 1545851258 1545851259

any ideas??

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Does each app require a different retention period for logging OR require different user access controls? If not then this should be in a single index, regardless if they are from different apps.

The solution DOES help if they are going from one index to another. The best way to think about this is, if you use the solution I suggested, your returning results from 3 different indexes into a single view. Index is irrelevant then at this point. Fields will be relative to sourcetype, so you will have identical fields across multiple indexes assuming your using the same sourcetype. You then join on that field which allows you to track the session.

If there's a particular order they come in, you can write the logic to account for that order. If not, then do it based off _time. You can structure each transaction relative to _time than coalesce them together to group the transactions

0 Karma

sriramfcam
New Member

How do i group them using coalesce

say my data is

12:00:00 app=A,
12:00:01 app=A
12:00:02 app=B
12:00:03 app=B
12:00:04 app=A
12:00:05 app=A,
12:00:06 app=C
12:00:07 app=C
12:00:08 app=A
12:00:09 app=A

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Coalesce will merge multiple fields values into a single field. You have 1 field called app with multiple values.. You should provide actual sample data if you want an accurate solution as you're not showing the whole picture here..

Another solution would be to add a unique guid to track the transactions.

0 Karma

sriramfcam
New Member

here is the real data (i've removed the actual app and guid names and moved all the calls into 1 index. The question now becomes how I show when the call enters and leaves each app

01/08/2019 16:49:11.045 app=A, guid=12345678, event=startAPI
01/08/2019 16:49:11.046 app=A, guid=12345678, event=endAPI
01/08/2019 16:49:11.047 app=B, guid=12345678, event=startAPI
01/08/2019 16:49:11.055 app=B, guid=12345678, event=endAPI
01/08/2019 16:49:11.056 app=A, guid=12345678, event=startAPI
01/08/2019 16:49:11.062 app=A, guid=12345678, event=endAPI
01/08/2019 16:49:11.063 app=C, guid=12345678, event=startAPI
01/08/2019 16:49:11.073 app=C, guid=12345678, event=endAPI
01/08/2019 16:49:11.074 app=A, guid=12345678, event=startAPI
01/08/2019 16:49:11.090 app=A, guid=12345678, event=endAPI

0 Karma

sriramfcam
New Member

Sorry.....prior data was wrong

01/08/2019 16:49:11.045 app=A, guid=12345678, event=startAPI
01/08/2019 16:49:11.046 app=A, guid=12345678, event=endAPI
01/08/2019 16:49:11.047 app=B, guid=12345678, event=startAPI
01/08/2019 16:49:11.055 app=B, guid=12345678, event=endAPI
01/08/2019 16:49:11.056 app=A, guid=12345678, event=startAPI
01/08/2019 16:49:11.062 app=A, guid=12345678, event=endAPI
01/08/2019 16:49:11.063 app=C, guid=12345678, event=startAPI
01/08/2019 16:49:11.073 app=C, guid=12345678, event=endAPI
01/08/2019 16:49:11.074 app=A, guid=12345678, event=startAPI
01/08/2019 16:49:11.090 app=A, guid=12345678, event=endAPI

and yes, retention period is the same

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...