topic Re: Regex in transaction endswith in Splunk Search

Help with Regex in transaction endswith

stwong — Tue, 04 Apr 2023 19:29:32 GMT

Hi all,

I try to group events using transaction. Since there are multiple endswith condition, i tried following to match either one of the 3 string patterns but unable to match:

... | transaction client endswith=eval(match(_raw, "string1|string2|string3"))

Would anyone please help?

Thanks a lot.

Best Rgds

Re: Regex in transaction endswith

javiergn — Tue, 04 Apr 2023 08:37:05 GMT

Hi @stwong ,

So far the only typos I've noticed in your regex are the missing parenthesis:

... | transaction client endswith=eval(match(_raw, "(string1|string2|string3)"))

If that's still not matching please post details about your raw data and the actual regex you are using.

Regards,

Re: Regex in transaction endswith

stwong — Tue, 04 Apr 2023 08:58:42 GMT

Hi @javiergn

Thanks for your help. Also tried adding parenthesis but still not working.

some sample data:

Mon Mar 27 15:38:53 2023 : Auth: (30281508) Login OK: [s123457@mydomain.hk] (from client proxy2 port 0 cli 1D6CE62B86E5) Mon Mar 27 15:38:53 2023 : Auth: (30281507) Login OK: [s123457@mydomain.hk] (from client proxy2 port 0 via TLS tunnel) Mon Mar 27 15:38:53 2023 : Auth: (30281447) Login incorrect (eap: rlm_eap (EAP): No EAP session matching state 0x6251908160558986): [plotter@connect.com.hk] (from client proxy1 port 0 cli 60AB67C12EFF) Mon Mar 27 15:38:53 2023 : Auth: (30281447) Login incorrect (Home Server says so): [plotter@connect.hku.hk] (from client proxy1 port 0 cli 50EB67C12EFA) Mon Mar 27 15:38:53 2023 : Auth: (30281488) Login OK: [s2345666@mydomain.hk] (from client proxy1 port 0 cli 3421D3826F64) Mon Mar 27 15:38:53 2023 : Auth: (30281487) Login OK: [s2345666@mydomain.hk] (from client proxy1 port 0 via TLS tunnel)

then search with

... | transaction client, host maxevents=2 maxpause=2 startswith="cli" endswith=eval(match(_raw, "(says|TLS)"))

the result is same if i remove the endswith setting which seems has no effect.

Thanks a lot.

Re: Regex in transaction endswith

javiergn — Tue, 04 Apr 2023 09:31:52 GMT

Hi,

I have uploaded your data to my lab and then extracted what I think are the host and client fields (correct me if I'm wrong please). Then I did the transaction as per the below:

index=foo source=example | rex "\[(?<host>[^\]]+).+client (?<client>\w+)" | transaction client, host maxevents=2 maxpause=2 startswith="cli" endswith=eval(match(_raw, "(says|TLS)")) | table client, host, _raw

And this is what I see as output, which looks OK to me based on the client and host:

If I remove the endswith from transaction this is what I get instead:

Which is very similar but not exactly the same. This is because you are forcing it to start with cli and then your maxevents are 2, meaning it's going to pick the next client/host combination matching the starting one. Whereas with the endswith you are forcing it to match says or TLS.

In any case, your event containing "says" won't match any other event in the list as the host (assumning I did the extraction correctly) is plotter@connect.hku.hk.

Let me know if that helps.

Re: Regex in transaction endswith

stwong — Thu, 06 Apr 2023 03:17:05 GMT

Hi,

Thanks a lot for your help.

Sorry that the host field is the log source host, to make sure we're looking at logs come from the same host.

It's strange that if the 2 lines match the criteria:

* client, host value

* startswith contains cli

* endswith contains says

but always can't be 'grouped' by transaction.

Mon Mar 27 15:38:53 2023 : Auth: (30281447) Login incorrect (eap: rlm_eap (EAP): No EAP session matching state 0x6251908160558986): [plotter@connect.com.hk] (from client proxy1 port 0 cli 60AB67C12EFF)
Mon Mar 27 15:38:53 2023 : Auth: (30281447) Login incorrect (Home Server says so): [plotter@connect.hku.hk] (from client proxy1 port 0 cli 50EB67C12EFA)

I uploaded file with these 2 lines only to test again. Tried different combinations by focusing on these 2 lines:

Not working:

startswith=eval(match(_raw, "(cli|eap)")) endswith="says" maxevents=2
startswith=eval(match(_raw, "(cli|eap)")) endswith=eval(match(_raw,"(says|TLS)")) maxevents=2

Can group into transaction:

startswith="eap" endswith=eval(match(_raw,"(says|TLS)")) maxevents=2
startswith="eap" endswith="says" maxevents=2

Did I miss anything?

Thanks a lot.

Best Regards

Re: Regex in transaction endswith

stwong — Thu, 06 Apr 2023 06:44:58 GMT

Hi,

Just tried this combination works:

| transaction client, host startswith=eval(match(_raw,"eap|OK:.*0\s+cli")) endswith=eval(match(_raw,"TLS|says")) maxevents=2

Seems the pattern has to be unambiguous among events to scan.
Not sure if my interpretation is correct.

Thanks a lot.

Best Rgds

Re: Help with Regex in transaction endswith

woodcock — Fri, 07 Apr 2023 00:24:36 GMT

The "transaction" command is one of the WORST scaling commands in all of splunk so it should never be used for a production use-case (because it fails without any indication and gives bad results). You should use "streamstats" instead (you can google this site for "woodcock correlationID" and get many examples that will get you there.