topic Re: How do I keep startswith from evicting a previous transaction? in Splunk Search

How do I keep startswith from evicting a previous transaction?

jswarren — Tue, 28 Jul 2015 22:35:21 GMT

Assume I have an input file like the following:

2015-07-28 12:00:01 Executing function a...
2015-07-28 12:00:02 debug1
2015-07-28 12:00:03 debug2
2015-07-28 12:00:04 Completing function a (value=-1)
2015-07-28 12:00:05 Executing function a...
2015-07-28 12:00:06 debug3
2015-07-28 12:00:07 debug4
2015-07-28 12:00:08 Completing function a (value=0)

I want to build a transaction object that begins with the first line and ends with the last. If I use startswith="Executing function a..." and endswith="Completing function a (value=0)", it appears that the second Executing function a... evicts the first, even though an endswith has not occurred. I end up with a single transaction that begins at 12:00:05 and ends at 12:00:08, and the earlier is an incomplete transaction.

Is there any way to get transaction not to evict the previous transaction when it encounters another startswith (prior to an endswith)?

Re: How do I keep startswith from evicting a previous transaction?

hgrow — Wed, 29 Jul 2015 12:14:29 GMT

Hi jswarren

I'm not sure if thats the key but your endswith="Completing function a (value=0)" is explicit looking for value=0.

It might be enough to simplify your search to something like:

... | transaction startswith="Executing" endswith="Completing"

sincerely
hgrow

Re: How do I keep startswith from evicting a previous transaction?

jswarren — Wed, 29 Jul 2015 13:46:21 GMT

If I do that, it results in two transactions, one from 12:00:01 - 12:00:04 and another from 12:00:05 - 12:00:08, which is not the desired outcome. The explicit "value=0" is requred.

Re: How do I keep startswith from evicting a previous transaction?

somesoni2 — Wed, 29 Jul 2015 14:59:07 GMT

Make sure that 'value' field is extracted, then try the search suggested by @hgrow and filter the incomplete result by checking "| where value=0". This will drop the transaction with value=-1 (incomplete)

Re: How do I keep startswith from evicting a previous transaction?

hgrow — Wed, 29 Jul 2015 15:05:49 GMT

Ah I see ... i've got that wrong. It's a tricky problem ...Im not sure if there is a simple way to not evict the first transaction.

If i get you right, you want all events from execution function a until Completing with value=0 in one transaction.It all depends on how your other events look like. Is it always function a? Are these events all in order? Maybe what you can try is to reduce your transaction to an endswith.
Something like ... | transaction endswith="(value=0)"

Re: How do I keep startswith from evicting a previous transaction?

jswarren — Wed, 29 Jul 2015 15:52:07 GMT

It's not the endswith that is the problem - I can successfully filter out the incorrect "ends". I can't filter out the incorrect "starts" because they are identical to each other. I need to somehow tell transaction to ignore repeated starts.

Re: How do I keep startswith from evicting a previous transaction?

jswarren — Wed, 29 Jul 2015 15:54:37 GMT

Answers to your questions:

No, it's not always "function a", it could be one of hundreds or thousands of different function names.
Yes, the events are in order and the process that writes the log is single-threaded.
I still have to have some way to identify the start of a transaction. What other options could I consider?

Re: How do I keep startswith from evicting a previous transaction?

emiller42 — Wed, 29 Jul 2015 17:59:37 GMT

Extract out the function name as a field, and use that in your transaction:

| rex "function\s(?<function>[^\s]+)\s" | transaction function startswith="Executing" endswith="(value=0)"

That should pull the whole shebang as a single transaction. However, this will omit any lines which do not have a 'function' field. (which may be context you need)

Re: How do I keep startswith from evicting a previous transaction?

emiller42 — Wed, 29 Jul 2015 18:06:15 GMT

If the logging is single-threaded (meaning you won't have processes interleaved with each other) then you can actually omit the 'startswith' and get what you want. Or, I do in testing at least.

If you may have multiple hosts running concurrently, then you want to include that in the transaction:

| transaction host endswith="(value=0)"

Re: How do I keep startswith from evicting a previous transaction?

jswarren — Wed, 29 Jul 2015 18:45:37 GMT

Ahh...yes...that would work, if there weren't "noise" between functions....I'll update the example to reflect that noise.

Re: How do I keep startswith from evicting a previous transaction?

jswarren — Wed, 29 Jul 2015 18:48:22 GMT

Or...I would post the update, if I had enough karma....

Assume that there are more events before the "Executing..." and after the "Completing" that should be excluded.

Re: How do I keep startswith from evicting a previous transaction?

emiller42 — Thu, 30 Jul 2015 14:32:44 GMT

Any way to filter out that noise? It might be a good idea to shape the initial search to only grab the lines you really care about from these transactions.