Why I am getting an additional Message level Error along with the Scheduled email alerts when the search condition are met?

[Hemnaath]

Hi All, Currently I am facing an issue in an scheduled email alert. We have scheduled a search query to trigger an email alert whenever splunk process goes down. Splunk query is working fine and we are also getting an alert whenever the conditions are met, but we are also getting Message level Error along with the result. Message level Error display all the indexer instance along with the error message.
Not sure why/what is causing to trigger additional message.

Query Details :

index=_internal host=hsp* OR host=vspl* sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log" "loader - Splunkd starting" OR "INFO ShutdownHandler - Shutting down splunkd" | eval message=if((message="Shutting down splunkd"),"failure","success") | rename message as status |eval Date_Time= readabledate + " " + readabletime | sort host | table host Date_Time status

Message Level: ERROR
1. [splunk01] Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'checkpoint:network:firewall' and lookup table 'checkpoint_vendor_info_lookup'.
2. [splunk02] Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'checkpoint:network:firewall' and lookup table 'checkpoint_vendor_info_lookup'.

Kindly guide me in how to fix this issue and where to start the troubleshooting steps.

thanks in advances.

[maciep]

Do you get those message level errors if you run the search manually?

If I had to guess, you have an automatic lookup configured somewhere called checkpoint_vendor_info_lookup. I'm not sure why it would be running for a search against the splunkd sourcetype, but maybe it's configured oddly. I would start there - find that lookup and see what it's doing and how it's configured.

A quick note about lookups. A lookup file is just a file on the server. A lookup definition references a lookup file and can add some properties around how the lookup will work (case sensitivity, wildcard, etc). And an automatic lookup will tell Splunk to automatically lookup fields in the specified lookup definition for any events from a given source, sourcetype or host.

[Hemnaath]

Hi Maciep, thanks for you're effort on this issue. We are not getting this message level error while running the query manually in search portal. Regarding automatic lookup is configured with the above name, but when checked the configuration via web, I could see sourcetype = empty filed under lookup inputs filed option in lookup table and also lookup output fields could see Product, vendor, vendor product= empty under lookup outputs filed. Kindly let me know how to trouble shoot this issue now.

thanks in advance.

[maciep]

When you run the search manually, are you in the same app where the scheduled report is?

If you disable the automatic lookup, does that resolve the issue? Maybe not the final solution, but would at least confirm that is the problem...

[Hemnaath]

hi maciep, once again thanks for you're effort, no i had run the query in different search head portal and more over last week we have upgraded the splunk environment from 6.2.1 to 6.6.1 series.

what is the purpose of using automatic lookup in splunk.

thanks in advance.

[maciep]

automatic lookups are just another way to enhance your data at search time. For example, let's say you have a sourcetype abc and that data has a message_id field. But that field is just a number you don't have all of the IDs memorized.

Well then you can build a lookup with the ID and Message Text. Then for sourcetype abc, you tell splunk to automatically lookup the id and spit out the message when searching that sourcetype.

The end result is that when searching for sourcetype abc, you'll now have the Message Text field as well, because it automatically doing that lookup for you. Hope that makes sense.

[Hemnaath]

thanks maciep for throwing some lights on automatic lookups. But how to fix this issue as I could see the Message level error being popped out whenever the saved search condition matched. In newly migrated clustered search head environment, could see the automatic lookup configured for this app. Kindly let me know how to fix this.

thanks in advance.

[maciep]

have you tried disabling the automatic lookup to verify that addresses the issue? If it does then you'd probably want to limit that lookup to a particular sourcetype (would probably require some changes to the local props.conf on your deployer for the app where the lookup is defined).

Assuming it's coming from some sort of checkpoint app you have installed, you could also reach out to the app developer for more questions about the lookup and what it's used for.

I mean, your search is looking directly in the splunkd sourcetype...it shouldn't be doing any checkpoint lookups.

[Hemnaath]

HI All, Can any one guide me on this issue.