IMAP app not working - debug gives python xml errors

[anewell]

I have a CentOS 5.5 x64 server with Splunk 4.1.6 and Dovecot 1.0.7-7. I'm not having any luck with getting the IMAP app to work, although manually IMAP'ing the mailbox does work.

Based on googling the debug output about "lmxl" I have installed the python-lxml package from repo, but that did not help. Server is a fairly plain CentOS install dedicated to Splunk.

Help?

[root@hostname default]# /opt/splunk/bin/python /opt/splunk/etc/apps/imap/bin/getimap.py --server=localhost --user=splunkmail --password=password --debug
DEBUG:root:END_IMAP_BREAKER=EndIMAPMessage
DEBUG:root:bodySourceType=imapbody
DEBUG:root:body_separator=____________________  Message Body  ____________________
DEBUG:root:debug=True
DEBUG:root:deleteWhenDone=False
DEBUG:root:folders=
DEBUG:root:fullHeaders=False
DEBUG:root:headerSourceType=imap
DEBUG:root:imapSearch=(UNDELETED SMALLER 204800)
DEBUG:root:includeBody=False
DEBUG:root:mimeTypes=text/plain
DEBUG:root:noCache=False
DEBUG:root:password=password
DEBUG:root:port=143
DEBUG:root:server=localhost
DEBUG:root:splunkHostPath=https://localhost:8089
DEBUG:root:splunkpassword=changeme
DEBUG:root:splunkuser=admin
DEBUG:root:splunkxpassword=
DEBUG:root:timeout=10
DEBUG:root:useBodySourceType=False
DEBUG:root:useSSL=False
DEBUG:root:user=splunkmail
DEBUG:root:version=1.2
DEBUG:root:xpassword=
DEBUG:root:VERSION = 1.2
DEBUG:root:about to dump mailbox *
DEBUG:root:/opt/splunk/lib/python2.6/site-packages/lxml/etree.so: symbol xmlSchematronSetValidStructuredErrors, version LIBXML2_2.6.32 not defined in file libxml2.so.2 with link time reference
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/imap/bin/getimap.py", line 405, in getMailbox
    latestTime = self.getCacheIDForMailbox(box)
  File "/opt/splunk/etc/apps/imap/bin/getimap.py", line 234, in getCacheIDForMailbox
    import splunk.auth as au
  File "/opt/splunk/lib/python2.6/site-packages/splunk/auth.py", line 1, in <module>
    import splunk, rest, util
  File "/opt/splunk/lib/python2.6/site-packages/splunk/rest/__init__.py", line 7, in <module>
    import lxml.etree as et
ImportError: /opt/splunk/lib/python2.6/site-packages/lxml/etree.so: symbol xmlSchematronSetValidStructuredErrors, version LIBXML2_2.6.32 not defined in file libxml2.so.2 with link time reference
None
ERROR:root:ERROR - trying to select mailgox

[root@hostname default]# telnet localhost 143
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
* OK Dovecot ready.
? login splunkmail password
? OK Logged in.
? LIST "" "*"
* LIST (\NoInferiors \UnMarked) "/" "INBOX"
? OK List completed.
? Select inbox
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
* 1 EXISTS
* 0 RECENT
* OK [UNSEEN 1] First unseen.
* OK [UIDVALIDITY 1297285491] UIDs valid
* OK [UIDNEXT 2] Predicted next UID
? OK [READ-WRITE] Select completed.
? select inbox
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
* 1 EXISTS
* 0 RECENT
* OK [UNSEEN 1] First unseen.
* OK [UIDVALIDITY 1297285491] UIDs valid
* OK [UIDNEXT 2] Predicted next UID
? OK [READ-WRITE] Select completed.
? select inbox
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
* 1 EXISTS
* 0 RECENT
* OK [UNSEEN 1] First unseen.
* OK [UIDVALIDITY 1297285491] UIDs valid
* OK [UIDNEXT 2] Predicted next UID
3 OK [READ-WRITE] Select completed.
? status inbox (messages)
* STATUS "inbox" (MESSAGES 1)
? OK Status completed.
? logout
* BYE Logging out
? OK Logout completed.
Connection closed by foreign host.

[root@hostname default]# rpm -qa | grep ^python
python-ctypes-1.0.2-1.1.el5
python-sqlite-1.1.7-1.2.1
python-iniparse-0.2.3-4.el5
python-urlgrabber-3.1.0-5.el5
python-ldap-2.2.0-2.1
python-smbios-2.2.19-10.1.el5
python-2.4.3-27.el5_5.3
python-lxml-1.3.4-1.el5.rf
python-numeric-23.7-2.2.2
python-elementtree-1.2.6-5

[Johnvey · Accepted solution]

The Splunk installer (for any platform) includes all the libraries necessary to import and use lxml. You should not install libxml* or lxml again within the Splunk environment. Splunk runs Python in its own sandbox, thus it is isolated from any Python binaries or modules that may already exist on your system.

If you would like to run arbitrary python scripts within the Splunk environment, use the following syntax:

$ /opt/splunk/bin/splunk cmd python PATH_TO_YOUR_SCRIPT

This loads the correct $PATH, and $PYTHONPATH variables so that all of the Splunk Python modules and libraries are available to the script.

(The IMAP app does not need to be manually executed -- the Splunk script runner is configured to automatically invoke this.)

[anewell]

Fixing the invocation of the test script helped. (grumble tab completion pebkac) App is running nicely from sched. search now.

Extraneous lxml package removed too.. Didn't think I needed it but was doing "due diligence". Not everyone is as reliable as Splunk at including dependencies!

Thank you!