Add-On Builder [pandas]: ModuleNotFoundError: No module named 'mmap'

[secret_lang]

Issue importing pandas module into Splunk Add-on Builder

import re
import sys
import os

ta_name = 'TA-urlparse'
ta_lib_name = '/usr/local/lib/python3.7/site-packages'
pattern = re.compile(r"[\\/]etc[\\/]apps[\\/][^\\/]+[\\/]bin[\\/]?$")
new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path]
new_paths.insert(0, os.path.sep.join([ta_lib_name]))
sys.path = new_paths

import numpy as np
import pandas
import requests
import ipaddress
import time
from datetime import datetime

 The Error:

Traceback (most recent call last):
  File "/apps/splunk/etc/apps/TA-urlparse/bin/urltest_1615256627_866.py", line 14, in <module>
    import input_module_urltest_1615256627_866 as input_module
  File "/apps/splunk/etc/apps/TA-urlparse/bin/input_module_urltest_1615256627_866.py", line 16, in <module>
    import pandas
  File "/usr/local/lib/python3.7/site-packages/pandas/__init__.py", line 52, in <module>
    from pandas.core.api import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/api.py", line 29, in <module>
    from pandas.core.groupby import Grouper, NamedAgg
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import DataFrameGroupBy, NamedAgg, SeriesGroupBy
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 57, in <module>
    from pandas.core.aggregation import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/aggregation.py", line 27, in <module>
    from pandas.core.series import FrameOrSeriesUnion, Series
  File "/usr/local/lib/python3.7/site-packages/pandas/core/series.py", line 68, in <module>
    from pandas.core import algorithms, base, generic, nanops, ops
  File "/usr/local/lib/python3.7/site-packages/pandas/core/generic.py", line 102, in <module>
    from pandas.io.formats import format as fmt
  File "/usr/local/lib/python3.7/site-packages/pandas/io/formats/format.py", line 71, in <module>
    from pandas.io.common import stringify_path
  File "/usr/local/lib/python3.7/site-packages/pandas/io/common.py", line 7, in <module>
    import mmap
ModuleNotFoundError: No module named 'mmap'

"Extensive" Troubleshooting:

/usr/bin/python3 -m pip install mmap
ERROR: Could not find a version that satisfies the requirement mmap
ERROR: No matching distribution found for mmap

Per: https://community.splunk.com/t5/All-Apps-and-Add-ons/how-can-I-enable-MMAP-caching-in-Google-GeoIP-o...

copied mmap.so and am now getting the following:

Traceback (most recent call last):
  File "/apps/splunk/etc/apps/TA-urlparse/bin/urltest_1615264087_827.py", line 14, in <module>
    import input_module_urltest_1615264087_827 as input_module
  File "/apps/splunk/etc/apps/TA-urlparse/bin/input_module_urltest_1615264087_827.py", line 16, in <module>
    import pandas
  File "/usr/local/lib/python3.7/site-packages/pandas/__init__.py", line 52, in <module>
    from pandas.core.api import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/api.py", line 29, in <module>
    from pandas.core.groupby import Grouper, NamedAgg
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import DataFrameGroupBy, NamedAgg, SeriesGroupBy
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 57, in <module>
    from pandas.core.aggregation import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/aggregation.py", line 27, in <module>
    from pandas.core.series import FrameOrSeriesUnion, Series
  File "/usr/local/lib/python3.7/site-packages/pandas/core/series.py", line 68, in <module>
    from pandas.core import algorithms, base, generic, nanops, ops
  File "/usr/local/lib/python3.7/site-packages/pandas/core/generic.py", line 102, in <module>
    from pandas.io.formats import format as fmt
  File "/usr/local/lib/python3.7/site-packages/pandas/io/formats/format.py", line 71, in <module>
    from pandas.io.common import stringify_path
  File "/usr/local/lib/python3.7/site-packages/pandas/io/common.py", line 7, in <module>
    import mmap
ImportError: /apps/splunk/lib/python3.7/mmap.so: undefined symbol: PL_mmap_page_size

Any Ideas? 

[mzvolsky]

In Python for Scientific Computing I see Py3.8 which differs from the version in splunk `bin/splunk cmd python --version`, which was 3.7.10. So I have Py3.7 installed (using `pyenv install 3.7.10` which enables more python versions on the machine) and from ~/.pyenv/versions/3.7.10 I have copied the both missing files into `/opt/splunk/lib/python3.7/lib-dynload`.

The `pip install splunk-sdk` then run well for me.

[mzvolsky]

to my previous answer (get missing mmap+cmath): .... the same can be sure achieved by selecting older version of `Py for Scientific computing` on the download page; but I have not an idea which one includes Py3.7.

[andreasz]

This is what worked for me: download the Python for Scientific Computing app https://splunkbase.splunk.com/app/2882 and copy the following two files 

/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python3.7/lib-dynload/mmap.cpython-37m-x86_64-linux-gnu.so

/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python3.7/lib-dynload/cmath.cpython-37m-x86_64-linux-gnu.so

to /opt/splunk/lib/python3.7

Tested it with custom search commands.