Splunk Enterprise

Add-On Builder [pandas]: ModuleNotFoundError: No module named 'mmap'

secret_lang
Observer

Issue importing pandas module into Splunk Add-on Builder

 

import re
import sys
import os

ta_name = 'TA-urlparse'
ta_lib_name = '/usr/local/lib/python3.7/site-packages'
pattern = re.compile(r"[\\/]etc[\\/]apps[\\/][^\\/]+[\\/]bin[\\/]?$")
new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path]
new_paths.insert(0, os.path.sep.join([ta_lib_name]))
sys.path = new_paths

import numpy as np
import pandas
import requests
import ipaddress
import time
from datetime import datetime

 

 The Error:

 

Traceback (most recent call last):
  File "/apps/splunk/etc/apps/TA-urlparse/bin/urltest_1615256627_866.py", line 14, in <module>
    import input_module_urltest_1615256627_866 as input_module
  File "/apps/splunk/etc/apps/TA-urlparse/bin/input_module_urltest_1615256627_866.py", line 16, in <module>
    import pandas
  File "/usr/local/lib/python3.7/site-packages/pandas/__init__.py", line 52, in <module>
    from pandas.core.api import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/api.py", line 29, in <module>
    from pandas.core.groupby import Grouper, NamedAgg
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import DataFrameGroupBy, NamedAgg, SeriesGroupBy
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 57, in <module>
    from pandas.core.aggregation import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/aggregation.py", line 27, in <module>
    from pandas.core.series import FrameOrSeriesUnion, Series
  File "/usr/local/lib/python3.7/site-packages/pandas/core/series.py", line 68, in <module>
    from pandas.core import algorithms, base, generic, nanops, ops
  File "/usr/local/lib/python3.7/site-packages/pandas/core/generic.py", line 102, in <module>
    from pandas.io.formats import format as fmt
  File "/usr/local/lib/python3.7/site-packages/pandas/io/formats/format.py", line 71, in <module>
    from pandas.io.common import stringify_path
  File "/usr/local/lib/python3.7/site-packages/pandas/io/common.py", line 7, in <module>
    import mmap
ModuleNotFoundError: No module named 'mmap'

 

"Extensive" Troubleshooting:

 

/usr/bin/python3 -m pip install mmap
ERROR: Could not find a version that satisfies the requirement mmap
ERROR: No matching distribution found for mmap

 

Per: https://community.splunk.com/t5/All-Apps-and-Add-ons/how-can-I-enable-MMAP-caching-in-Google-GeoIP-o...

copied mmap.so and am now getting the following:

Traceback (most recent call last):
  File "/apps/splunk/etc/apps/TA-urlparse/bin/urltest_1615264087_827.py", line 14, in <module>
    import input_module_urltest_1615264087_827 as input_module
  File "/apps/splunk/etc/apps/TA-urlparse/bin/input_module_urltest_1615264087_827.py", line 16, in <module>
    import pandas
  File "/usr/local/lib/python3.7/site-packages/pandas/__init__.py", line 52, in <module>
    from pandas.core.api import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/api.py", line 29, in <module>
    from pandas.core.groupby import Grouper, NamedAgg
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import DataFrameGroupBy, NamedAgg, SeriesGroupBy
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 57, in <module>
    from pandas.core.aggregation import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/aggregation.py", line 27, in <module>
    from pandas.core.series import FrameOrSeriesUnion, Series
  File "/usr/local/lib/python3.7/site-packages/pandas/core/series.py", line 68, in <module>
    from pandas.core import algorithms, base, generic, nanops, ops
  File "/usr/local/lib/python3.7/site-packages/pandas/core/generic.py", line 102, in <module>
    from pandas.io.formats import format as fmt
  File "/usr/local/lib/python3.7/site-packages/pandas/io/formats/format.py", line 71, in <module>
    from pandas.io.common import stringify_path
  File "/usr/local/lib/python3.7/site-packages/pandas/io/common.py", line 7, in <module>
    import mmap
ImportError: /apps/splunk/lib/python3.7/mmap.so: undefined symbol: PL_mmap_page_size

Any Ideas? 

Labels (3)
Tags (1)
0 Karma

mzvolsky
Engager

In Python for Scientific Computing I see Py3.8 which differs from the version in splunk `bin/splunk cmd python --version`, which was 3.7.10. So I have Py3.7 installed (using `pyenv install 3.7.10` which enables more python versions on the machine) and from ~/.pyenv/versions/3.7.10 I have copied the both missing files into `/opt/splunk/lib/python3.7/lib-dynload`.

The `pip install splunk-sdk` then run well for me.

0 Karma

mzvolsky
Engager

to my previous answer (get missing mmap+cmath): .... the same can be sure achieved by selecting older version of `Py for Scientific computing` on the download page; but I have not an idea which one includes Py3.7.

0 Karma

andreasz
Path Finder

This is what worked for me: download the Python for Scientific Computing app https://splunkbase.splunk.com/app/2882 and copy the following two files 

/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python3.7/lib-dynload/mmap.cpython-37m-x86_64-linux-gnu.so
/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python3.7/lib-dynload/cmath.cpython-37m-x86_64-linux-gnu.so
 
to /opt/splunk/lib/python3.7
 
Tested it with custom search commands.
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...