Splunk Search

Change outputlookup to events

jwhughes58
Contributor

Hi All,

The Bloodhound TA creates a KV store lookup.  I've been asked to take the entries in the KV store and turn them into events.  I've setup an alert, but I'm not seeing the alert fire.  The SPL looks like this

 

| inputlookup path_principals_lookup 
| eval domain_id=if(isnull(domain_id), "NULL_domain_id", domain_id) 
| eval domain_name=if(isnull(domain_name), "NULL_domain_name", domain_name) 
| eval group=if(isnull(group), "NULL_Group", group) 
| eval non_tier_zero_principal=if(isnull(non_tier_zero_principal), "NULL_non_tier_zero_principal", non_tier_zero_principal) 
| eval path_id=if(isnull(path_id), "NULL_path_id", path_id) 
| eval path_title=if(isnull(path_title), "NULL_path_title", path_title) 
| eval principal=if(isnull(principal), "NULL_principal", principal) 
| eval tier_zero_principal=if(isnull(tier_zero_principal), "NULL_tier_zero_principal", tier_zero_principal) 
| eval user=if(isnull(user), "NULL_user", user) 
| dedup domain_id, domain_name, group, non_tier_zero_principal, path_id, path_title, principal, tier_zero_principal, user

 

I see statistics, but that doesn't fire the alert.  Is there something I'm missing to turn the values in the kvstore into events to be alerted on?

TIA,

Joe

Labels (2)
Tags (2)
0 Karma
1 Solution

dtburrows3
Builder

Just a follow up, are you readjusting the cron schedule to fire soon after making the adjustment to test?
I'm not sure if an alert action will trigger by just doing an "Open in Search" or "Run" action from the UI. I think the scheduler may have to kick off the search for the alert actions to be applied. (unless using the "| sendalert command")

View solution in original post

dtburrows3
Builder

When you say you have set up an alert, what are your configured Trigger Conditions and then the following Alert Actions that follow?
These can be found in the Edit Alert menu and looks like this.

dtburrows3_0-1704481778641.png


Where does the outputlookup come into play here? I dont see it in you SPL shared but it is in the title.
From just the title of this question alone it sounds like you would like to gather results and instead of storing them in a lookup to send them to a summary index via alert_action or collect command.

But from the body of the question it sounds like you are just having issues seeing results of a scheduled search trigger an alert. If you run the search ad-hoc and are seeing results, then I would check Trigger conditions, the configured alert actions if the trigger conditions are met. 
If those look good then I would check the ownership of the Alert itself and does the owner have access to the KVStore.

You should be able to look into internal logs about the status of previous runs as well with something like this.

index=_internal savedsearch_name="<alert_name>"
    | table _time, savedsearch_name, user, app, status, dispatch_time, run_time, result_count, alert_actions, action_time_ms

where <alert_name> is the name of your alert.

I also noticed on your search that you had a lot of eval doing sort of the same function, I think a foreach loop might be useful here if you want to try it out.

| inputlookup path_principals_lookup
    | foreach domain_id, domain_name, group, non_tier_zero_principal, path_id, path_title, principal, tier_zero_principal, user
        [
            | eval
                <<FIELD>>=if(
                    isnull('<<FIELD>>'), 
                        "NULL_<<FIELD>>", 
                        '<<FIELD>>'
                    )
            ]
    | dedup domain_id, domain_name, group, non_tier_zero_principal, path_id, path_title, principal, tier_zero_principal, user
0 Karma

jwhughes58
Contributor

Cool.  Not quite as fast as the original method, but the difference is minuscule.  I do like the fact that I don't have to repeat the same command.  This is nice to know.

0 Karma

jwhughes58
Contributor

The outputlookup should have been inputlookup.  My brain slipped a gear when I was entering the Subject.  I have corrected it.  Here is what I have in the alert.  I should give the foreach a try.

0 Karma

dtburrows3
Builder

Okay so I think since your trigger condition is 

 

 

search count>0

 

 

It suspect is not firing because there is no field named 'count' for that to evaluate as true.

Can you try this setting instead? (it should be the same logic as intended)

dtburrows3_0-1704486100885.png

As long as the KVStore has results in it, then your alert action should trigger every time the scheduler kick off the search.

Tags (1)
0 Karma

jwhughes58
Contributor

I've tried that and I didn't see anything.  I tried it again and I still don't see the alert firing.

0 Karma

dtburrows3
Builder

Just a follow up, are you readjusting the cron schedule to fire soon after making the adjustment to test?
I'm not sure if an alert action will trigger by just doing an "Open in Search" or "Run" action from the UI. I think the scheduler may have to kick off the search for the alert actions to be applied. (unless using the "| sendalert command")

jwhughes58
Contributor

I just changed the cron job.  I was just running it from the UI.   Once I did that, I started getting alerts.  I need to do some more cleanup, but the problem is solved. 

0 Karma

dtburrows3
Builder

Awesome! Glad you got it resolved!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...