Splunk Enterprise

Add-On Builder [pandas]: ModuleNotFoundError: No module named 'mmap'

secret_lang
Observer

Issue importing pandas module into Splunk Add-on Builder

 

import re
import sys
import os

ta_name = 'TA-urlparse'
ta_lib_name = '/usr/local/lib/python3.7/site-packages'
pattern = re.compile(r"[\\/]etc[\\/]apps[\\/][^\\/]+[\\/]bin[\\/]?$")
new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path]
new_paths.insert(0, os.path.sep.join([ta_lib_name]))
sys.path = new_paths

import numpy as np
import pandas
import requests
import ipaddress
import time
from datetime import datetime

 

 The Error:

 

Traceback (most recent call last):
  File "/apps/splunk/etc/apps/TA-urlparse/bin/urltest_1615256627_866.py", line 14, in <module>
    import input_module_urltest_1615256627_866 as input_module
  File "/apps/splunk/etc/apps/TA-urlparse/bin/input_module_urltest_1615256627_866.py", line 16, in <module>
    import pandas
  File "/usr/local/lib/python3.7/site-packages/pandas/__init__.py", line 52, in <module>
    from pandas.core.api import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/api.py", line 29, in <module>
    from pandas.core.groupby import Grouper, NamedAgg
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import DataFrameGroupBy, NamedAgg, SeriesGroupBy
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 57, in <module>
    from pandas.core.aggregation import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/aggregation.py", line 27, in <module>
    from pandas.core.series import FrameOrSeriesUnion, Series
  File "/usr/local/lib/python3.7/site-packages/pandas/core/series.py", line 68, in <module>
    from pandas.core import algorithms, base, generic, nanops, ops
  File "/usr/local/lib/python3.7/site-packages/pandas/core/generic.py", line 102, in <module>
    from pandas.io.formats import format as fmt
  File "/usr/local/lib/python3.7/site-packages/pandas/io/formats/format.py", line 71, in <module>
    from pandas.io.common import stringify_path
  File "/usr/local/lib/python3.7/site-packages/pandas/io/common.py", line 7, in <module>
    import mmap
ModuleNotFoundError: No module named 'mmap'

 

"Extensive" Troubleshooting:

 

/usr/bin/python3 -m pip install mmap
ERROR: Could not find a version that satisfies the requirement mmap
ERROR: No matching distribution found for mmap

 

Per: https://community.splunk.com/t5/All-Apps-and-Add-ons/how-can-I-enable-MMAP-caching-in-Google-GeoIP-o...

copied mmap.so and am now getting the following:

Traceback (most recent call last):
  File "/apps/splunk/etc/apps/TA-urlparse/bin/urltest_1615264087_827.py", line 14, in <module>
    import input_module_urltest_1615264087_827 as input_module
  File "/apps/splunk/etc/apps/TA-urlparse/bin/input_module_urltest_1615264087_827.py", line 16, in <module>
    import pandas
  File "/usr/local/lib/python3.7/site-packages/pandas/__init__.py", line 52, in <module>
    from pandas.core.api import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/api.py", line 29, in <module>
    from pandas.core.groupby import Grouper, NamedAgg
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import DataFrameGroupBy, NamedAgg, SeriesGroupBy
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 57, in <module>
    from pandas.core.aggregation import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/aggregation.py", line 27, in <module>
    from pandas.core.series import FrameOrSeriesUnion, Series
  File "/usr/local/lib/python3.7/site-packages/pandas/core/series.py", line 68, in <module>
    from pandas.core import algorithms, base, generic, nanops, ops
  File "/usr/local/lib/python3.7/site-packages/pandas/core/generic.py", line 102, in <module>
    from pandas.io.formats import format as fmt
  File "/usr/local/lib/python3.7/site-packages/pandas/io/formats/format.py", line 71, in <module>
    from pandas.io.common import stringify_path
  File "/usr/local/lib/python3.7/site-packages/pandas/io/common.py", line 7, in <module>
    import mmap
ImportError: /apps/splunk/lib/python3.7/mmap.so: undefined symbol: PL_mmap_page_size

Any Ideas? 

Labels (3)
Tags (1)
0 Karma

mzvolsky
Engager

In Python for Scientific Computing I see Py3.8 which differs from the version in splunk `bin/splunk cmd python --version`, which was 3.7.10. So I have Py3.7 installed (using `pyenv install 3.7.10` which enables more python versions on the machine) and from ~/.pyenv/versions/3.7.10 I have copied the both missing files into `/opt/splunk/lib/python3.7/lib-dynload`.

The `pip install splunk-sdk` then run well for me.

0 Karma

mzvolsky
Engager

to my previous answer (get missing mmap+cmath): .... the same can be sure achieved by selecting older version of `Py for Scientific computing` on the download page; but I have not an idea which one includes Py3.7.

0 Karma

andreasz
Path Finder

This is what worked for me: download the Python for Scientific Computing app https://splunkbase.splunk.com/app/2882 and copy the following two files 

/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python3.7/lib-dynload/mmap.cpython-37m-x86_64-linux-gnu.so
/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python3.7/lib-dynload/cmath.cpython-37m-x86_64-linux-gnu.so
 
to /opt/splunk/lib/python3.7
 
Tested it with custom search commands.
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...