External lookup - Splunk UI can't see all binaries in the folder

[nquba]

I have two python scripts for external lookup. Both of them use two different binaries under location /home/xxx/bin64/. My first lookup works perfectly fine. When I wrote my second lookup the splunk UI was showing me error - 'No such file or directory' -

To troubleshoot this issue, I wrote a simple python script that just list all the files in the folder /home/xxx/bin64 and ran it on the host as:

splunk cmd python test.py

This showed me all the files perfectly fine. But when I ran a similar lookup that just takes clientip and output list of files under /home/xxx/bin64 in lscheck field as

index=test sourcetype=stest | lookup testlookup clientip as ip | table lscheck 

The lscheck field do list files, but the list is not complete - there are four files that are missing. I don't understand why Splunk UI is unable to see all the files, though the file permissions are same?

Thank you

[nquba · Accepted solution]

This was identified to be as a bug - Thanks

[goelli]

Hi,
Is this bug resolved? If yes, in which version?
Thanks.

[Richfez]

What four files does it not list? The first four, last four, all that end with .sh, the middle four or perhaps a random-seeming selection? Does it always skip the SAME four or does it miss different ones different times? How many files are there in there?

How is the script actually reading the list of files? If you run the script manually (perhaps redirecting output or something), it displays all the files? If you output that list to a file, then try to ingest that resulting file, does that read all correctly or does it miss those four as well?

Also, you write near the beginning that when you wrote the second lookup the Splunk UI showed you the error "No such file or directory". When did it show you that (like, at what point in creating the lookup?)

If the scripts are short and simple enough, pasting them in may help, too.

[nquba]

What four files does it not list?
The files it does not see include.

htmltree
jwhois
mech-dump
xpath

The first four, last four
Well this depends on how you sort them, name, modification time, inode, etc.. I don't find any pattern.

Does it always skip the SAME four?
Yes it always skip the SAME four. For testing I added another binary called test and it did not appear either

If the scripts are short and simple enough, pasting them in may help, too.
It is a python script, very similar to existing external_lookup.py with little tweaks to add.

The original script was for doing whoislookup, but when it did not work I wrote simple test scripts as below:

vi ls_lookup.py

#!/usr/bin/env python
from subprocess import PIPE, Popen
import os
import csv
import sys
import json

def listfiles(ip):

        try:
          p1 = Popen(["ls", "/home/xxx/bin64"],stdout=PIPE,stderr=PIPE)
          (out,err) = p1.communicate()
          if out:
            return out
          else:
            return str(err)
        except Exception as e:
          return e

def main():

        if len(sys.argv) !=3:
          print "Usage: python ls_lookup.py [ip_field] [ls_output]"
          sys.exit(1)
        ip_field = sys.argv[1]
        ls_output = sys.argv[2]

        infile = sys.stdin
        outfile = sys.stdout

        r = csv.DictReader(infile)
        header = r.fieldnames
        w = csv.DictWriter(outfile, fieldnames=r.fieldnames)
        w.writeheader()

        for result in r:
          if result[ip_field] and result[ls_output]:
            w.writerow(result)

          if result[ip_field]:
            result[ls_output] = listfiles(result[ip_field])
            w.writerow(result)

main()

Test command:

printf "clientip,lscheck\n10.0.0.0n" | /splunk/bin/python ls_lookup.py  clientip lscheck

The above test command list all the files perfectly fine.

Now when I do this in the Splunk UI:

index=test sourcetype=stest | lookup ls_lookup clientip as  ip | table ip, lscheck

In the lscheck field I see list of files, but it does not list the above mentioned four files 😞

"No such file or directory"
If I change the popen in above script to "p1 = Popen(["ls", "/home/xxx/bin64/jwhois"],stdout=PIPE,stderr=PIPE)", it gives an error 'No such file or directory' in the lscheck field, for obvious reason 🙂

[nquba]

In my second test I listed all the files under $SPLUNK/etc/system/bin/, partial result is pasted below

     ip                  lscheck

       10.x.x.y          addeventtype.py addtail.py appinstall.py appsmanager.py apptemplates.py datamodel DataModelHandlerUtil.pyc external_lookup.py field_extractor.py localapps.py pdfgen_endpoint_is_available.py pdfgen_endpoint.py sc_rest.py sc_rest.pyc surrounding_events.py


         10.x.x.z      addeventtype.py addtail.py appinstall.py appsmanager.py apptemplates.py external_lookup.py field_extractor.py localapps.py pdfgen_endpoint_is_available.py pdfgen_endpoint.py sc_rest.py surrounding_events.py


         10.x.x.2    addeventtype.py addtail.py appinstall.py appsmanager.py apptemplates.py datamodel DataModelHandlerUtil.pyc external_lookup.py field_extractor.py localapps.py pdfgen_endpoint_is_available.py pdfgen_endpoint.py sc_rest.py sc_rest.pyc surrounding_events.py

Note two things:

1- The second event has less number of files listed than the first and third events (datamodel is missing) have same number of files.
2- In all of the three events, my ls_lookup.py file is missing, though it is there. All the other additional files are missing. So Splunk UI is looking only the files that came with it..

I also tried adding the line [fschange:$SPLUNK/etc/system/bin/] in the inputs.conf file, restarted the splunk, but did not help..