Update Macro From Search

[Thulasinathan_M]

Hi,

I've a case where I want to update/append the Macro with the results from lookup. I don't want to do this manually each time. So is there any way I could use a scheduled search and update macro if the lookup has any new values.

[ITWhisperer]

If there were a way to update a macro, it would likely to have a ReST endpoint, but there doesn't appear to be one. Having said that, even if there were, this sounds like a risky thing to be doing anyway. Perhaps a better way would be to update a lookup or kv store with the results from your search so that the macro can use those i.e. keep the processing (defined by the macro) separate from the data (found by the search). What you seem to be asking for smacks of self-modifying code, which, while it may sound like a cool thing to do, is generally not a safe practice.

[Thulasinathan_M]

Thanks @ITWhisperer for your valuable info. My lookup has full of rex Patterns (1000s of patterns), but I don't want to dump this in a macro. That's why thought to update macro only if I start seeing new Patterns in the result event. If you could help me with this specific use-case it would be very much helpful. Thanks in advance.

[ITWhisperer]

Does your look up have 1000s of patterns or your macro has 1000s of patterns or both?

Where do these patterns come from?

Please explain with a bit more detail and examples what your usecase is?

[Thulasinathan_M]

I have 1000s of rex Patterns which is already available in a lookup file, but I don't want to put everything into macro. So I thought to update macro only if I start seeing events match any of rex pattern in lookup but not in macro. So by doing this I have minimal rex pattern in macro (For now I've 232 rex patterns in macro).

[Thulasinathan_M]

Let's say below are few rex Patterns available in my lookup

| rex field=LogLine mode=sed "s|(Deprecated configuration detected in path).*( Please update your settings to use the latest configuration options.)|\1 \2|g"
| rex field=LogLine mode=sed "s|(Query execution time exceeded the threshold:).*(seconds. Query: SELECT * FROM users WHERE last_login).*|\1 \2|g"
| rex field=LogLine mode=sed "s|(Query execution time exceeded the threshold:).*(seconds. Query: SELECT * FROM contacts WHERE contact_id).*|\1 \2|g"

Below are the search results, I want to use above rex Pattern:

WARN  ConfigurationLoader - Deprecated configuration detected in path /xx/yy/zz. Please update your settings to use the latest configuration options.
WARN  ConfigurationLoader - Deprecated configuration detected in path /aa/dd/jkl. Please update your settings to use the latest configuration options.
WARN  QueryExecutor - Query execution time exceeded the threshold: 12.3 seconds. Query: SELECT * FROM users WHERE last_login > '2024-01-01'.
WARN  QueryExecutor - Query execution time exceeded the threshold: 21.9 seconds. Query: SELECT * FROM contacts WHERE contact_id > '252’.

 
So I'll get something like below, if I do stats

LogLine	Count
Deprecated configuration detected in path . Please update your settings to use the latest configuration options.	2
Query execution time exceeded the threshold: seconds. Query: SELECT * FROM users WHERE last_login	1
Query execution time exceeded the threshold: seconds. Query: SELECT * FROM contacts WHERE contact_id	1

[ITWhisperer · Accepted solution]

Convert your lookup so it has a pattern and name for the pattern e.g.

logline	pattern
Deprecated configuration detected in path Please update your settings to use the latest configuration options.	*Deprecated configuration detected in path* Please update your settings to use the latest configuration options.*
Query execution time exceeded the threshold: seconds. Query: SELECT * FROM users WHERE last_login	*Query execution time exceeded the threshold:*seconds. Query: SELECT * FROM users WHERE last_login*
Query execution time exceeded the threshold: seconds. Query: SELECT * FROM contacts WHERE contact_id	*Query execution time exceeded the threshold:*seconds. Query: SELECT * FROM contacts WHERE contact_id*

Then add a lookup definition and use advanced option to set WILDCARD(pattern)

Now you can use lookup on your events to find out which type of loglines you have

| lookup patterns.csv pattern as _raw
| stats count by logline

[Thulasinathan_M]

Thank you, @ITWhisperer. It's working as expected 😊

[ITWhisperer]

So let me see if I have understood:

You have 1000s of patterns in a lookup which you use against a set of events and if any of the events match against a pattern in the lookup you copy that pattern into a macro? And this is the process you want to automate?

[Thulasinathan_M]

Absolutely, correct. That's my intention and I'm bit worried if I would hit a Performance impact if I keep on updating the macro and it exceeds limit at some point. Is there any better approach I can deal with for this use-case. Happy to adapt to any better approaches.