I've started to see the following messages from some of my forwarding instances of splunk:
11-30-2010 16:50:02.355 ERROR LookupOperator - The lookup table 'sqlagent_jobs' does not exist. It is referenced by configuration 'mssql_processes'.
This error is somewhat correct. I do have "sqlagent_jobs" defined in
transforms.conf, but the
sqlagent_jobs.csv is missing on the forwarder's app. What I'm trying to figure out is why the forwarder cares.
I'm familiar with getting these
LookupOperator errors on the search-head (and interactively) when I forget part of the lookup configuration or have some kind of permissions issue--but I thought that was strictly caused by running a search. On my forwarders, there are no running searches. (I use heavy-weight forwarders, so it's possible to run search on them; but there's no locally indexed data to search; and I see no entries in
scheduler.log so I don't think that's the case, but I could be missing something.)
Does anyone know what else would trigger this error?
(As a work around, I'm commenting out the
LOOKUP-* entries in my deployment-apps app for the moment, but I don't want to have to maintain two versions of my app if I don't have too. I use the same app on the search-head and on my forwarders)
I'm guessing there are actually searches being run on your heavyweight forwarders. By default I believe we have scheduled searches that run that populate various status dashboards and it may be these searches that are triggering the error.
I think I disabled all the of internal searches for this very reason. But supposing I missed one, can you think of any reasons why I'm not seeing anything logged in
scheduler.log (both of these are 0 bytes on the forwarder),
audit.log has nothing about running searches, and there's nothing in
dispatch folder. Is there anything else I should look for?