topic Re: Transaction to Streamstats in Splunk Search

Transaction to Streamstats

tb5821 — Tue, 01 Sep 2020 14:45:29 GMT

I'm trying to move from using a transaction command to a streamstats - I get most of the way there but I can't figure out the 'reset' with streamstats to proper group starts and ends together. Here's my current query which isn't finding the right end time since my BY query will match multiple places and the latest(_time) will change to the wrong latest time but yet the actual latest time ...

index="logfiles" appType=reports* "Generating * Status Report" | rex \.(?<reportName>Generate\w*) | eval reportName=replace (reportName,"Generate","") | eval reportName=replace (reportName,"Report","") | streamstats earliest(_time) as stime by reportName appType | join appType,reportName [search index="logfiles" appType=reports* "Report generated successfully" | rex \.(?<reportName>\w+)ReportGenerator | streamstats reset_on_change=true reset_after="("searchmatch(\"Report generated successfully\")")" latest(_time) as etime BY reportName appType] | eval diff=etime-stime | eval hhmmss=tostring(diff, "duration") | convert timeformat=" %a %b %d %I:%M:%S.%3N %p %Z" ctime(stime) as StartTime | table StartTime appType reportName hhmmss |rename hhmmss as RunDuration

Re: Transaction to Streamstats

tb5821 — Fri, 04 Sep 2020 16:34:59 GMT

anyone?

Re: Transaction to Streamstats

ITWhisperer — Fri, 04 Sep 2020 17:19:52 GMT

@tb5821 Do you have some sample events you can share to show what you are dealing with?

Re: Transaction to Streamstats

tb5821 — Sat, 05 Sep 2020 16:09:06 GMT

the events are long and span hours if not close to a day... but the beginning of the event starts with the generating line and ends with a completed log line. The problem is that if server 1 picks up job A and runs for 28hrs. Then server 2 will pick up job A 12 hours into server 1's run. This is fine b/c they are different servers. BUT now the issue comes in ~24hours after server1 starts is that its time for the next run, and server 1 again on a different thread picks up that run. So its the same server1 and same job A but its start and end should be different.

Yet whats happening is my streamstats for the first event computes the event end as the end for the last (latest) run.

Re: Transaction to Streamstats

ITWhisperer — Sun, 06 Sep 2020 08:48:53 GMT

Hi @tb5821

By events I meant splunk event e.g. log entries. What is the data you are trying to process with splunk?

Re: Transaction to Streamstats

niketn — Sun, 06 Sep 2020 17:26:25 GMT

@tb5821 for the community to assist you better please provide more details like some sample (cooked up event, should have masking and anonymization for any sensitive data). Also provide your current SPL with transaction command which is also doing the job (but I am guessing it is not performing).