Building for the Splunk Platform

Modular input that need additional python modules

phoenixdigital
Builder

Hi All,

So I am trying to make a modular input which ingests the journal in newer LInux distributions which no longer log to a file.

I have this test Python script which works from the command line.

import select
import systemd
from systemd import journal

j = systemd.journal.Reader()
#j.log_level(journal.LOG_DEBUG)

# j.add_match(_SYSTEMD_UNIT="systemd-udevd.service")
j.seek_tail()

# need to get previous event as seek_tail went right to the very end of the journal
j.get_previous()

p = select.poll()
p.register(j, j.get_events())

while p.poll():
    if j.process() != journal.APPEND:
        continue

    for entry in j:
        event = str(entry['__REALTIME_TIMESTAMP'])
        for key, value in entry.iteritems():
            event = event + ',' + str(key) + '="' + str(value).replace('"','\\"') + '"' 

        print event

Now the issue I have is when I try to add it to a modular input script. Splunk's Python obviously doesn't have the library for systemd and I get this error

[root@phoenix bin]# /opt/splunk/bin/splunk cmd /opt/splunk/bin/python /opt/splunk/etc/apps/journal_input/bin/journal_input.py
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/journal_input/bin/journal_input.py", line 20, in <module>
    from systemd import journal
ImportError: No module named systemd

Everything I have read says to leave Splunk's python alone and not to try to install custom modules.

Does anyone have any suggestions how I can get this into a modular input or to get systemd accessible to Splunk's Python binary without having support freak out?

Tags (2)
0 Karma
1 Solution

musskopf
Builder

Hi, have a look on my comment on this topic: http://answers.splunk.com/answers/109009/splunk6-django-app-importing-sqlite.html

I basically download the source of same version of Python used by Splunk and compiled modules outsite... after that I moved the compiled modules to the Splunk/lib/python dir.

View solution in original post

musskopf
Builder

Hi, have a look on my comment on this topic: http://answers.splunk.com/answers/109009/splunk6-django-app-importing-sqlite.html

I basically download the source of same version of Python used by Splunk and compiled modules outsite... after that I moved the compiled modules to the Splunk/lib/python dir.

phoenixdigital
Builder

Thanks this has definitely stopped the importing of the library issue.

cp -r /usr/lib64/python2.7/site-packages/systemd /opt/splunk/lib/python2.7/site-packages/
chown -R splunk:users /opt/splunk/lib/python2.7/site-packages/

I did want to try to avoid this however as I know Splunk officially does not recommend this.

I note in your thread Splunk were looking into resolving the need for this in 2003. It seems they moved onto more important issues since.

0 Karma

musskopf
Builder

There will be always the need to additional packages if you developed custom search/lookup scripts. I would prefer Splunk to ship the Python source/dev packages + virtualenv so would be easier to just run pip install PKG_NAME 🙂

0 Karma

phoenixdigital
Builder

Agreed it would make things a hell of a lot easier. I am guessing they don't want to support tweaking the Python libraries and in a way I can see why.

Sadly it really limits the power of what we can achieve with modular and scripted inputs though.

0 Karma
Get Updates on the Splunk Community!

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Admin Console: A Single, Unified Interface for All Your Cloud Admin Needs

WATCH NOWJoin us to learn how the admin console can save you time and give you more control over the Splunk® ...