Multiprocessing funcion "Pool()" cannot be used in my lookup external python script on Linux.

[Suda]

Hello,

I cannot use one of multiprocessing functions, "Pool()" in my lookup external python script on CentOS 6.3 with Splunk 5.0.1.

On the other hand,
I can use the same python script Splunk 5.0.1 on Windows 7.

And I can also use "Pool()" on the same machine with "/usr/bin/python" (version 2.7.3).


Then I believe my machine can support python "Pool()" function, and my lookup python script would be no problem in point of grammer and functions.

If you have any workaround solution to use "Pool()" in lookup external scripts, could you kindly explain me?

I'd like to share my investigated resutls with you.

---

Here is my "test.py".

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    pool = Pool(processes=4)
    result = pool.apply_async(f, [10])
    print result.get(timeout=1)
    print pool.map(f, range(10))

---

Here is my results on Splunk's python with the following command.

/opt/splunk/bin/splunk cmd python test.py

Traceback (most recent call last):File "test.py", line 7, in <module>
    pool = Pool(processes=4)
  File "/opt/splunk/lib/python2.7/multiprocessing/__init__.py", line 232, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "/opt/splunk/lib/python2.7/multiprocessing/pool.py", line 115, in __init__
    self._setup_queues()
  File "/opt/splunk/lib/python2.7/multiprocessing/pool.py", line 209, in _setup_queues
    from .queues import SimpleQueue
  File "/opt/splunk/lib/python2.7/multiprocessing/queues.py", line 48, in <module>
    from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition
  File "/opt/splunk/lib/python2.7/multiprocessing/synchronize.py", line 59, in <module>
    " function, see issue 3770.")
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

---

Here is also resuls on /usr/bin/python and Splunk on Windows 7 with the following command.

"C:\Program files\Splunk\bin\splunk.exe" cmd python test.py

100
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

---

I guess...

When python was build in Splunk, some build parameters would be different from general build parameters.

I'm not sure why they changed build parameters???

Thank you.

[Suda]

I confirm this issue has been fixed before Splunk 6.0.3. I cannot reproduce the issue with Splunk 6.0.3 on CentOS.

[Drainy · Accepted solution]

Not quite, its a bug that affects a couple of Python versions and it would appear that the build of Python included with Splunk also includes this bug. It may be worth raising a support case to see if they can include a fix in a maintenance release.

Looking around the net it seems to be a bit intermittent on which versions and which platforms it can occur on but it does seem to at least be a known bug.

Bug report: http://bugs.python.org/issue3770

[Suda]

Thank you for your answer.
I reported this case to Splunk support.