topic Re: Can you do a data model search based on tstats and macros? in Splunk Search

Can you do a data model search based on tstats and macros?

david_casey — Mon, 06 Aug 2018 13:53:29 GMT

Can you do a data model search based on a macro? Trying but Splunk is not liking it. It yells about the wildcards *, or returns no data depending on different syntax.

Base data model search:

| tstats summariesonly count FROM datamodel=Web.Web BY Web.url, Web.client_ip

The macro (coinminers_url) contains url patterns as follows:

"0aqpqdju.me" OR "0x1f4b0.com" OR "12finance.com" OR "1480876790.rsc.cdn77.org" OR "1beb2a44.space" OR "1q2w3.fun" OR "1q2w3.me"

The objective is to cross reference the macro for any matches in the Web data model and display the date/time and internal client IP that hit one (or more) matches from the macro list. These will be the clients that hit a known coinminer domain, which we maintain a list of based on a GitHub repo list. That list is quite large so a traditional search can take quite a while.

Suggestions?

Re: Can you do a data model search based on tstats and macros?

DalJeanis — Mon, 06 Aug 2018 14:24:31 GMT

You can build a macro that will use the WHERE fieldname IN ("list","of","values") format. Per the docs, the below format should work with wildcards in the macro. https://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/ConditionalFunctions

First try this with 3 values and see if it works...

 | tstats summariesonly count FROM datamodel=Web.Web BY Web.url, Web.client_ip 
    where Web.url IN ("0aqpqdju.me", "0x1f4b0.com",...)

If it works, then just build a macro that resolves to that whole second line.

... if you can't get the above working, then try this, but it will not work with wildcards...

 | tstats summariesonly count FROM datamodel=Web.Web BY Web.url, Web.client_ip 
 | where in(Web.url ,"0aqpqdju.me", "0x1f4b0.com",...)

Re: Can you do a data model search based on tstats and macros?

david_casey — Mon, 06 Aug 2018 15:03:15 GMT

Thanks D... No go unfortunately.

| tstats summariesonly count FROM datamodel=Web.Web BY Web.url, Web.client_ip where Web.url IN ("0aqpqdju.me", "0x1f4b0.com",...)

Error: Error in 'TsidxStats': Wildcards (*) are not supported in groupby fields

(FYI - The wildcards are being stripped in the above query when posted here.)

As to the second option it won't work. It has to contain the wildcards since the url can vary wildly.

Seems like this should be a simple pattern matching search against the DM.

Someone suggested using macros.conf and transforms.conf in another channel. Will give that a try.