Lookup table does not exist from indexer but everything is global

[madhack]

I've configured a CSV lookup and an automatic lookup on Splunk 5.0.4 that work on one of my search heads (let's call it host01). When I push the app to the indexer search peer (host02) that holds the data, host01 starts showing errors about the lookup not existing:

[host02] The lookup table 'internal_domains' does not exist. It is referenced by configuration 'source::maillog|host::mailhost|sendmail_syslog'.

All of my searching has led me to believe this kind of thing is normally a permission issue on any of the pieces involved (lookup table file, lookup defintion, or automatic lookup) but the ONLY "*.meta" files I can find that contain any information about this lookup on my indexer are in my app, and it has this:

[props]
export = system

[lookups/internal_domains.csv]
export = system
version = 5.0.2
modtime = 1367367795.814840000
access = read : [ * ], write : [ admin, power ]
owner = nobody

[transforms/internal_domains]
export = system
version = 5.0.3
access = read : [ * ], write : [ admin, power ]
modtime = 1371773947.230195000
owner = nobody

[props/sendmail_syslog/LOOKUP-direction]
access = read : [ * ], write : [ admin, power ]
owner = nobody
version = 5.0.4
modtime = 1378938232.175058000

The most confusing part is that if I log in to host02 and do the exact same search, I don't get any errors and the automatic lookup happens, regardless of what app I do it from. Meanwhile, the errors didn't start showing up on host01 until I'd pushed the definitions to host02. I'm sure I must be missing something obvious.

[lguinn2]

It is my understanding that there should be no references to the lookup table in your indexer's configuration files. Lookups should be defined on the search head, and that is also where the lookup tables are stored. Distributed search takes care of distributing the lookups to the indexer as needed.

There is one caveat though - the lookup (file, definition, and automatic lookup definition) should NOT be private. They should be consistent (as you noted) and have permissions at either the app or global level.

I would

1. Remove the csv file, the props.conf entries and the transforms.conf entries from host2

2. Make sure that all these items exist on host1, with permissions of app or global

3. The .csv file should have the same ownership and permissions as the various .conf files

4. Check that you have set up distributed search on host1

[madhack]

There are users who only have access to host02 and not host01 for various reasons. I only index on host02 but I search on both host01 and host02. Am I to understand that this is not a supported configuration?

[madhack]

The splunk user owns it. That first bit in UNIX permissions refers to the owner.

splunk@evgconlnx06:~/etc/apps/euc$ head -1 lookups/internal_domains.csv
domain,is_internal

I can read the file as the splunk user, and I can perform the lookup manually and automatically under any situation as long as I'm doing it on the local search head and not across a search peer. File permissions aren't the issue, I'm afraid...

[lukejadamec]

I'm not a big Linux guy, but from what I know with a 600 it should not work anywhere. Only root can read it.

[madhack]

There isn't any Windows involved; I should have specified that both hosts are Linux. The CSV was created using a Python script on another Linux box and scped over.

The file is on both hosts owned by splunk:splunk mode 600. The app was pushed via deployment server.

splunk@host01:~/etc/apps/euc$ ls -l lookups/internal_domains.csv
-rw------- 1 splunk splunk 3073 Sep 10 22:48 lookups/internal_domains.csv

splunk@host02:~/etc/apps/euc$ ls -l lookups/internal_domains.csv
-rw------- 1 splunk splunk 3073 Sep 11 22:24 lookups/internal_domains.csv

[lukejadamec]

Windows does not transfer permissions well. Did you check the Windows access rights for the .csv file on host02 and host01?
How did you "push" the app from host01 to host02?

Does the .csv still exist on host01?