topic Re: How to accelerate using transaction in Splunk Search

How to accelerate using transaction

keekkenen — Mon, 16 Jul 2018 09:28:05 GMT

Hi, all

for example, I want find all transactions contains some word. How to make it more faster ?
If I have too much transaction my search work too long.

For example, my search look like
source="wms"
| transaction process startswith="START" endswith="END"
| search platzToRepack

where process some extracted fields (based on regexp)

Please, recommend me some strategies for faster searching

Re: How to accelerate using transaction

niketn — Mon, 16 Jul 2018 11:37:25 GMT

@keekkenen can you add some sample events for process START and process END? Also what is platzToRepack filter? Can it not be added to base search instead of being present after the transaction command?

Ideally you can try using stats instead of transaction. There are several examples on Splunk Answers. For example https://answers.splunk.com/answers/511699/how-to-get-the-number-of-records-of-a-field-and-su.html.
However, for your use case please provide more data so that we can provide something more specific.

Following is Splunk Documentation for choosing between various event correlation commands in Splunk:
http://docs.splunk.com/Documentation/Splunk/latest/Search/Abouteventcorrelation

Re: How to accelerate using transaction

niketn — Mon, 16 Jul 2018 11:37:58 GMT

Following is Splunk Documentation for choosing between various event correlation commands in Splunk:
http://docs.splunk.com/Documentation/Splunk/latest/Search/Abouteventcorrelation

Re: How to accelerate using transaction

gcusello — Mon, 16 Jul 2018 11:41:19 GMT

Hi keekkenen,
the transaction command is a very slow command for its own nature, for this reason it's better to use other solutions, when possible.
At first, the events in your transaction are only the ones that contain "START" and "END" or there are more events?
if you want to take only the events that contain "START" and "END", you could use stats command

source="wms" (START OR END) 
| stats values(my_field1) AS my_field1 values(my_field2) AS my_field2 earliest(_time) AS _time
| search platzToRepack
| table _time my_field1 my_field2

If instead you have a common field, you could run something like this

source="wms" common_field=*
| stats values(my_field1) AS my_field1 values(my_field2) AS my_field2 earliest(_time) AS _time BY common_field
| search platzToRepack
| table _time common_field my_field1 my_field2

Otherwise, the only way to improve performances of transaction command is to reduce the time using maxspan option when possible.

Bye.
Giuseppe

Re: How to accelerate using transaction

keekkenen — Mon, 16 Jul 2018 13:41:14 GMT

sample events
2018-07-12 21:39:44,979 [WCS-11 ] ice.impl.DbHandlerService$DbWorkerThread INFO START (some process..)
2018-07-12 21:39:44,980 [WCS-11 ] .wms.server.manager.nutzer.NutzerManager TRACE about something 1
2018-07-12 21:39:44,981 [WCS-11 ] server.manager.system.SystemRolleManager TRACE about something 2
2018-07-12 21:39:44,981 [WCS-11 ] g.wms.server.manager.nutzer.RechtManager TRACE about something 3
2018-07-12 21:39:44,983 [WCS-11 ] g.wms.server.manager.nutzer.RechtManager TRACE about something etc
2018-07-12 21:40:45,973 [WCS-11 ] ice.impl.DbHandlerService$DbWorkerThread INFO END (end process..)

I want found transaction by process (for example, WCS-11), where one or more messages contains some word(s) (for example, 'about something' or 'platzToRepack' )

Re: How to accelerate using transaction

keekkenen — Mon, 16 Jul 2018 13:48:23 GMT

No, events has any content, and I want get a transaction - first event with 'START" word plus all events between it and next event (last event in transaction) with 'END' word (see sample events in post above)

Re: How to accelerate using transaction

keekkenen — Tue, 17 Jul 2018 15:28:30 GMT

I found few solution for my case, but it work unreal too long..
for example, in log file by 1 hour period I have about 4 million events, it contains more 98 thousand transactions (there are really fat transactions - may be about 600 events), and, for example, in it transactions only one transaction what I try found

for find it transaction need 25 min, for check contains it transaction in events need 4 min, it's really too long
I found it transaction in total commander more quickly