Why is alert_action value is 0 or empty but the actual log has "email" value?

[vn_g]

Query :

index=_internal sourcetype=scheduler thread_id="AlertNotifier*" "email"
| fillnull

Sample Log :

12-06-2020 08:05:11.189 +0000 INFO SavedSplunker - savedsearch_id="nobody;search;TimeDelayInCiscoEstreamerLogsFromHost-"10.90.78.45"", search_type="", user="abc", app="search", savedsearch_name="TimeDelayInCiscoEstreamerLogsFromHost-"10.90.78.45"", priority=default, status=success, digest_mode=1, scheduled_time=1607241900, window_time=0, dispatch_time=1607241903, run_time=6.113, result_count=2, alert_actions="email", sid="scheduler_dmFybNyaS5hZG1pbkBiY2cuY29t__search__RMD5e388bf8114eaecc6_at_160900_26417_8B358556-EC52-4F41-A194-1A98CFD37560", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""

Issue : Actual log has alert_actions="email" value , when checked in selected fields the value is either 0 or empty even after using fillnull. What does this mean? In which scenarios does this occur? My use case is to find out the number of email alerts triggered in the last 24 hours , but few of them are missing in my report because alert_actions is either showing 0 or empty value.

[scelikok]

@vn_g, the problem seems on source field. Splunk source field for scheduler logs in not "scheduler". It contains full path. Can you please try below query?

index=_internal sourcetype=scheduler alert_actions="*email*"

OR

index=_internal source=*scheduler.log alert_actions="*email*"

[vn_g]

I tried both the above queries, I still have the same issue.

[scelikok]

So, I agree with @harsmarvania57 , you should check your extractions props.conf and transforms.conf for "scheduler" sourcetype.

[harsmarvania57]

Which version of splunk? I am on Splunk 7.2 and I can see alert_actions field populated with correct data.

Can you please try to run below search and check whether you get any result?

index=_internal sourcetype=scheduler alert_actions="*email*"

EDIT: My mistake instead of source it should be sourcetype in above query which I corrected. Great catch @scelikok 

[vn_g]

Version : 8.0.4.1

Query gives below result :index=_internal sourcetype=scheduler "email"
| fillnull

12-06-2020 08:05:11.189 +0000 INFO SavedSplunker - savedsearch_id="nobody;search;TimeDelayInCiscoEstreamerLogsFromHost-"10.90.78.45"", search_type="", user="abc", app="search", savedsearch_name="TimeDelayInCiscoEstreamerLogsFromHost-"10.90.78.45"", priority=default, status=success, digest_mode=1, scheduled_time=1607241900, window_time=0, dispatch_time=1607241903, run_time=6.113, result_count=2, alert_actions="email", sid="scheduler_dmFybNyaS5hZG1pbkBiY2cuY29t__search__RMD5e388bf8114eaecc6_at_160900_26417_8B358556-EC52-4F41-A194-1A98CFD37560", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""

Query : index=_internal source=scheduler alert_actions="*email*"
Above query doesnt give any results.

[harsmarvania57 · Accepted solution]

For me, query which I have provided working fine in 8.0.4.1 . Do you have any field extractions for field name alert_actions, if it is there then it is overriding default extractions. Check props.conf and transforms.conf for field alert_actions.

[vn_g]

Splunk was not auto extracting "alert_actions" field using normal key value pair extractions.

App: search
props.conf
[scheduler]
EXTRACT-alert_action = alert_actions\=\"(?P<alert_actions>.*?)\"

Applying above props solved the issue.

[harsmarvania57]

Hi,

alert_actions="" means there was no actions performed when scheduled search completed. For example: No email alert action, no summary indexing etc.

[vn_g]

Yes, but in my case i could see alert_actions="email" in the actual log but when the field and the value are being shown in bottom of the log , because alert_actions is in selected fields , it shows 0 value or doesnt display it because of null value.