topic Modular input that need additional python modules in Splunk Dev

Modular input that need additional python modules

phoenixdigital — Tue, 17 Mar 2015 03:58:52 GMT

Hi All,

So I am trying to make a modular input which ingests the journal in newer LInux distributions which no longer log to a file.

I have this test Python script which works from the command line.

import select
import systemd
from systemd import journal

j = systemd.journal.Reader()
#j.log_level(journal.LOG_DEBUG)

# j.add_match(_SYSTEMD_UNIT="systemd-udevd.service")
j.seek_tail()

# need to get previous event as seek_tail went right to the very end of the journal
j.get_previous()

p = select.poll()
p.register(j, j.get_events())

while p.poll():
    if j.process() != journal.APPEND:
        continue

    for entry in j:
        event = str(entry['__REALTIME_TIMESTAMP'])
        for key, value in entry.iteritems():
            event = event + ',' + str(key) + '="' + str(value).replace('"','\\"') + '"' 

        print event

Now the issue I have is when I try to add it to a modular input script. Splunk's Python obviously doesn't have the library for systemd and I get this error

[root@phoenix bin]# /opt/splunk/bin/splunk cmd /opt/splunk/bin/python /opt/splunk/etc/apps/journal_input/bin/journal_input.py
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/journal_input/bin/journal_input.py", line 20, in <module>
    from systemd import journal
ImportError: No module named systemd

Everything I have read says to leave Splunk's python alone and not to try to install custom modules.

Does anyone have any suggestions how I can get this into a modular input or to get systemd accessible to Splunk's Python binary without having support freak out?

Re: Modular input that need additional python modules

musskopf — Tue, 17 Mar 2015 04:32:23 GMT

Hi, have a look on my comment on this topic: http://answers.splunk.com/answers/109009/splunk6-django-app-importing-sqlite.html

I basically download the source of same version of Python used by Splunk and compiled modules outsite... after that I moved the compiled modules to the Splunk/lib/python dir.

Re: Modular input that need additional python modules

phoenixdigital — Tue, 17 Mar 2015 23:38:52 GMT

Thanks this has definitely stopped the importing of the library issue.

cp -r /usr/lib64/python2.7/site-packages/systemd /opt/splunk/lib/python2.7/site-packages/
chown -R splunk:users /opt/splunk/lib/python2.7/site-packages/

I did want to try to avoid this however as I know Splunk officially does not recommend this.

I note in your thread Splunk were looking into resolving the need for this in 2003. It seems they moved onto more important issues since.

Re: Modular input that need additional python modules

musskopf — Tue, 17 Mar 2015 23:58:31 GMT

There will be always the need to additional packages if you developed custom search/lookup scripts. I would prefer Splunk to ship the Python source/dev packages + virtualenv so would be easier to just run pip install PKG_NAME 🙂

Re: Modular input that need additional python modules

phoenixdigital — Wed, 18 Mar 2015 01:01:31 GMT

Agreed it would make things a hell of a lot easier. I am guessing they don't want to support tweaking the Python libraries and in a way I can see why.

Sadly it really limits the power of what we can achieve with modular and scripted inputs though.