topic Re: How can I group and aggregate and filter all events based on those aggregated attributes? in Splunk Search

How can I group and aggregate and filter all events based on those aggregated attributes?

cjoelly — Tue, 18 Jun 2024 11:47:34 GMT

Coming from SQL, I want to do stuff like GROUP BY and HAVING ...

The data is available with a transaction identifier.Grouing should be done by that transaction identifier.

Per transaction, I want to check a few attributes, if their values are unique within each treansaction.

In SQL terms:
select transaction_id from index group by transaction_id

having count(distinct attr1) = 1 and count(distinct attr2) = 1 and count(distinct attr3) = 1

From that table of transaction_ids, a join to the same index should be done to filter the events.

How can I achieve this with Splunk query?

Re: How can I group and aggregate and filter all events based on those aggregated attributes?

ITWhisperer — Tue, 18 Jun 2024 12:37:16 GMT

You could try using eventstats to tag each event with the aggregated value for the transaction it is a part of and use this to filter the events.

Re: How can I group and aggregate and filter all events based on those aggregated attributes?

cjoelly — Tue, 18 Jun 2024 14:04:54 GMT

I dont know if my approach is the right way to go. As I learned, that JOINs allow only 50.000 records to be joined. And I expect way more events to be joined to the filtered transactions.

Re: How can I group and aggregate and filter all events based on those aggregated attributes?

cjoelly — Wed, 19 Jun 2024 06:45:13 GMT

We changed our approach, generate a different structure in Splunk using stats and thus we do not need to read the raw events anymore.