topic Re: Update Macro From Search in Splunk Search

Update Macro From Search

Thulasinathan_M — Fri, 13 Sep 2024 07:19:55 GMT

Hi,

I've a case where I want to update/append the Macro with the results from lookup. I don't want to do this manually each time. So is there any way I could use a scheduled search and update macro if the lookup has any new values.

Re: Update Macro From Search

ITWhisperer — Fri, 13 Sep 2024 07:34:53 GMT

If there were a way to update a macro, it would likely to have a ReST endpoint, but there doesn't appear to be one. Having said that, even if there were, this sounds like a risky thing to be doing anyway. Perhaps a better way would be to update a lookup or kv store with the results from your search so that the macro can use those i.e. keep the processing (defined by the macro) separate from the data (found by the search). What you seem to be asking for smacks of self-modifying code, which, while it may sound like a cool thing to do, is generally not a safe practice.

Re: Update Macro From Search

Thulasinathan_M — Fri, 13 Sep 2024 08:10:44 GMT

Thanks @ITWhisperer for your valuable info. My lookup has full of rex Patterns (1000s of patterns), but I don't want to dump this in a macro. That's why thought to update macro only if I start seeing new Patterns in the result event. If you could help me with this specific use-case it would be very much helpful. Thanks in advance.

Re: Update Macro From Search

ITWhisperer — Fri, 13 Sep 2024 08:29:19 GMT

Does your look up have 1000s of patterns or your macro has 1000s of patterns or both?

Where do these patterns come from?

Please explain with a bit more detail and examples what your usecase is?

Re: Update Macro From Search

Thulasinathan_M — Fri, 13 Sep 2024 08:37:31 GMT

I have 1000s of rex Patterns which is already available in a lookup file, but I don't want to put everything into macro. So I thought to update macro only if I start seeing events match any of rex pattern in lookup but not in macro. So by doing this I have minimal rex pattern in macro (For now I've 232 rex patterns in macro).

Re: Update Macro From Search

ITWhisperer — Fri, 13 Sep 2024 08:57:30 GMT

So let me see if I have understood:

You have 1000s of patterns in a lookup which you use against a set of events and if any of the events match against a pattern in the lookup you copy that pattern into a macro? And this is the process you want to automate?

Re: Update Macro From Search

Thulasinathan_M — Fri, 13 Sep 2024 08:58:23 GMT

Let's say below are few rex Patterns available in my lookup

| rex field=LogLine mode=sed "s|(Deprecated configuration detected in path).*( Please update your settings to use the latest configuration options.)|\1 \2|g" | rex field=LogLine mode=sed "s|(Query execution time exceeded the threshold:).*(seconds. Query: SELECT * FROM users WHERE last_login).*|\1 \2|g" | rex field=LogLine mode=sed "s|(Query execution time exceeded the threshold:).*(seconds. Query: SELECT * FROM contacts WHERE contact_id).*|\1 \2|g"

Below are the search results, I want to use above rex Pattern:

WARN ConfigurationLoader - Deprecated configuration detected in path /xx/yy/zz. Please update your settings to use the latest configuration options. WARN ConfigurationLoader - Deprecated configuration detected in path /aa/dd/jkl. Please update your settings to use the latest configuration options. WARN QueryExecutor - Query execution time exceeded the threshold: 12.3 seconds. Query: SELECT * FROM users WHERE last_login > '2024-01-01'. WARN QueryExecutor - Query execution time exceeded the threshold: 21.9 seconds. Query: SELECT * FROM contacts WHERE contact_id > '252’.

So I'll get something like below, if I do stats

LogLine	Count
Deprecated configuration detected in path . Please update your settings to use the latest configuration options.	2
Query execution time exceeded the threshold: seconds. Query: SELECT * FROM users WHERE last_login	1
Query execution time exceeded the threshold: seconds. Query: SELECT * FROM contacts WHERE contact_id	1

Re: Update Macro From Search

Thulasinathan_M — Fri, 13 Sep 2024 09:01:06 GMT

Absolutely, correct. That's my intention and I'm bit worried if I would hit a Performance impact if I keep on updating the macro and it exceeds limit at some point. Is there any better approach I can deal with for this use-case. Happy to adapt to any better approaches.

Re: Update Macro From Search

ITWhisperer — Fri, 13 Sep 2024 10:15:07 GMT

Convert your lookup so it has a pattern and name for the pattern e.g.

logline	pattern
Deprecated configuration detected in path Please update your settings to use the latest configuration options.	Deprecated configuration detected in path Please update your settings to use the latest configuration options.*
Query execution time exceeded the threshold: seconds. Query: SELECT * FROM users WHERE last_login	Query execution time exceeded the threshold:seconds. Query: SELECT * FROM users WHERE last_login*
Query execution time exceeded the threshold: seconds. Query: SELECT * FROM contacts WHERE contact_id	Query execution time exceeded the threshold:seconds. Query: SELECT * FROM contacts WHERE contact_id*

Then add a lookup definition and use advanced option to set WILDCARD(pattern)

Now you can use lookup on your events to find out which type of loglines you have

| lookup patterns.csv pattern as _raw | stats count by logline

Re: Update Macro From Search

Thulasinathan_M — Fri, 13 Sep 2024 11:28:59 GMT

Thank you, @ITWhisperer. It's working as expected 😊