topic Re: StartsWith breaks the transaction in Splunk Search

StartsWith breaks the transaction

gmasy — Wed, 01 Apr 2020 08:34:14 GMT

Hello everyone,

I am trying to extract some data from the logs.
I have created a little search that works well:

customergetservice host=MBKBKKSPHTRSP0* source="/var/log/jbossas/standalone/server.log" | transaction RequestId

It links the request and the response from the server by the extracted field RequestId.

However, I am trying to filter only some parameter contained in the field Login:

The login has many possibilities of different values, but it starts with either:

"BBC-*"
"BBF-*"
"BFL-*"
"BCIMR-*"

The login is only in the request, and not in the response.
I am trying to filter by putting the following

customergetservice host=MBKBKKSPHTRSP0* source="/var/log/jbossas/standalone/server.log" | transaction RequestId startswith="BCIMR-*"

However, when I add the StartsWith, everything's broken, as the events are not linked together anymore.
See following comment for example as I can't add more than 2 images per question

Re: StartsWith breaks the transaction

gmasy — Wed, 01 Apr 2020 08:36:18 GMT

If someone has any input it'd be greatly appreciated.
Thank you

Re: StartsWith breaks the transaction

PavelP — Wed, 01 Apr 2020 12:01:41 GMT

try to add match/searchmatch to startswith:

customergetservice host=MBKBKKSPHTRSP0* source="/var/log/jbossas/standalone/server.log" | transaction RequestId startswith=eval(searchmatch("BCIMR-"))

Re: StartsWith breaks the transaction

to4kawa — Wed, 01 Apr 2020 12:43:30 GMT

customergetservice host=MBKBKKSPHTRSP0* source="/var/log/jbossas/standalone/server.log"  (login="BBC-*" OR login="BBF-*" OR login="BFL-*" OR login="BCIMR-*") OR "LEAN-RESPONSE"
| rex "(?<status>LEAN-(REQUEST|RESPONSE))"
| stats dc(status) as flag values(_raw) as _raw values(_time) as _time by RequestId
| where flag > 1

you don't need transaction

transaction command consumes much search time.
stats is almost useful and no problem.
because stats makes events to combine one event by same field value.
Isn't it the same as the transaction command?

my previous answer has a few typo(statsu). I fix it.

Is there the login field? please try again.

Re: StartsWith breaks the transaction

woodcock — Wed, 01 Apr 2020 14:43:22 GMT

Ditch transaction; it is overkill and does not scale well; try this:

index="YouShouldAlwaysSepcifyIndex" AND sourcetype="AndSourcetypeToo" AND "customergetservice" AND host="MBKBKKSPHTRSP0*" AND source="/var/log/jbossas/standalone/server.log"
| rex "LEAN-(?<status>REQUEST|RESPONSE)"
| reverse
| streamstats count(eval(searchmatch("BCIMR-*"))) AS TransactionID BY RequestId
| stats range(_time) AS duration count AS eventcount values(status) AS status dc(status) AS statuscount min(_time) AS _time BY TransactionID RequestId

Re: StartsWith breaks the transaction

gmasy — Wed, 01 Apr 2020 17:51:34 GMT

Thanks to the three of you for your quick input about my problem!
I'm happily surprised of the reactivity on this forum, as this was my very first post...

However, none of the solutions provided did work for me.
@PavelP, although your solution seemed great, I got the same result as attached in previous screenshot. The response and request are not grouped in the same event when adding the startswith ...

@to4kawa and @woodcock , thank you for your suggestions, could you please explain me why transaction is overkill for what I am trying to achieve?
It's the only solution I have found in the Splunk documentation to match two corresponding events in the same splunk result.
It enables me to see what is in each request and what is the associated response.
stats here is not the thing I am trying to achieve just yet.

Thank you

Re: StartsWith breaks the transaction

to4kawa — Wed, 01 Apr 2020 20:22:15 GMT

I see. my answer is updated. please check and try.

Re: StartsWith breaks the transaction

gmasy — Thu, 02 Apr 2020 02:35:12 GMT

This one actually worked like a charm and is 10 times faster than the transaction method!
Thank you so much

Is there a way to do a timechart of the duration between the request and its response with the solution you provided?

Re: StartsWith breaks the transaction

to4kawa — Thu, 02 Apr 2020 02:59:27 GMT

a timechart of the duration between the request and its response
What's x-axis and y-axis?

There is no keep-alive log. Your log has only REQUEST and RESPONSE.

Everybody wants to make time chart. but It's very hard.
Good luck.

Re: StartsWith breaks the transaction

gmasy — Thu, 02 Apr 2020 03:10:48 GMT

x-axis would be the time
y-axis would be te average response time

We like to monitor the response time over a period, to detect if a component is down.

Please see example below of what I could achieve with transaction. However this method is very long and usually times out if I check on a very long time span.

Re: StartsWith breaks the transaction

to4kawa — Thu, 02 Apr 2020 04:15:06 GMT

 customergetservice host=MBKBKKSPHTRSP0* source="/var/log/jbossas/standalone/server.log"  (login="BBC-*" OR login="BBF-*" OR login="BFL-*" OR login="BCIMR-*") OR "LEAN-RESPONSE"
 | rex "(?<status>LEAN-(REQUEST|RESPONSE))"
 | stats dc(status) as flag min(_time) as _time range(_time) as duration by RequestId
 | where flag > 1
 | fields _time RequestId duration
 | timechart span=1h avg(duration)

If you want to calculate by each RequestId, use by clause.