We upgraded our indexers from 6.6.4 to 7.3.3 and now any search gives us:
[sptsp005] Could not load lookup=LOOKUP-simple_identity_lookup
[sptsp006] Could not load lookup=LOOKUP-simple_identity_lookup
We used btools to dig into the automatic lookup and that exists and works.
The odd part this message is referencing the indexers. Has anyone ever seen this error point to the indexers instead of the search
head? I backed up /etc/ before the upgrade and nothing really looks different or overwritten.
Is the lookup actually present on the indexers? Most Apps/Some TAs are only installed on the search head, meaning if the lookup came that way it won't be on the indexers and you'll get a warn at search time. By default, Splunk will attempt to use lookups on the indexers, as that spreads the load out and it's one less thing for the search head to do when results come in. However, you can if you wish set local=true as part of the search which invokes the lookup so that it's not run on the peers. Take a look here:
https://docs.splunk.com/Documentation/Splunk/8.0.1/SearchReference/Lookup#Optional_arguments