Splunk Enterprise

Add-On Builder [pandas]: ModuleNotFoundError: No module named 'mmap'

secret_lang
Observer

Issue importing pandas module into Splunk Add-on Builder

 

import re
import sys
import os

ta_name = 'TA-urlparse'
ta_lib_name = '/usr/local/lib/python3.7/site-packages'
pattern = re.compile(r"[\\/]etc[\\/]apps[\\/][^\\/]+[\\/]bin[\\/]?$")
new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path]
new_paths.insert(0, os.path.sep.join([ta_lib_name]))
sys.path = new_paths

import numpy as np
import pandas
import requests
import ipaddress
import time
from datetime import datetime

 

 The Error:

 

Traceback (most recent call last):
  File "/apps/splunk/etc/apps/TA-urlparse/bin/urltest_1615256627_866.py", line 14, in <module>
    import input_module_urltest_1615256627_866 as input_module
  File "/apps/splunk/etc/apps/TA-urlparse/bin/input_module_urltest_1615256627_866.py", line 16, in <module>
    import pandas
  File "/usr/local/lib/python3.7/site-packages/pandas/__init__.py", line 52, in <module>
    from pandas.core.api import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/api.py", line 29, in <module>
    from pandas.core.groupby import Grouper, NamedAgg
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import DataFrameGroupBy, NamedAgg, SeriesGroupBy
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 57, in <module>
    from pandas.core.aggregation import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/aggregation.py", line 27, in <module>
    from pandas.core.series import FrameOrSeriesUnion, Series
  File "/usr/local/lib/python3.7/site-packages/pandas/core/series.py", line 68, in <module>
    from pandas.core import algorithms, base, generic, nanops, ops
  File "/usr/local/lib/python3.7/site-packages/pandas/core/generic.py", line 102, in <module>
    from pandas.io.formats import format as fmt
  File "/usr/local/lib/python3.7/site-packages/pandas/io/formats/format.py", line 71, in <module>
    from pandas.io.common import stringify_path
  File "/usr/local/lib/python3.7/site-packages/pandas/io/common.py", line 7, in <module>
    import mmap
ModuleNotFoundError: No module named 'mmap'

 

"Extensive" Troubleshooting:

 

/usr/bin/python3 -m pip install mmap
ERROR: Could not find a version that satisfies the requirement mmap
ERROR: No matching distribution found for mmap

 

Per: https://community.splunk.com/t5/All-Apps-and-Add-ons/how-can-I-enable-MMAP-caching-in-Google-GeoIP-o...

copied mmap.so and am now getting the following:

Traceback (most recent call last):
  File "/apps/splunk/etc/apps/TA-urlparse/bin/urltest_1615264087_827.py", line 14, in <module>
    import input_module_urltest_1615264087_827 as input_module
  File "/apps/splunk/etc/apps/TA-urlparse/bin/input_module_urltest_1615264087_827.py", line 16, in <module>
    import pandas
  File "/usr/local/lib/python3.7/site-packages/pandas/__init__.py", line 52, in <module>
    from pandas.core.api import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/api.py", line 29, in <module>
    from pandas.core.groupby import Grouper, NamedAgg
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import DataFrameGroupBy, NamedAgg, SeriesGroupBy
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 57, in <module>
    from pandas.core.aggregation import (
  File "/usr/local/lib/python3.7/site-packages/pandas/core/aggregation.py", line 27, in <module>
    from pandas.core.series import FrameOrSeriesUnion, Series
  File "/usr/local/lib/python3.7/site-packages/pandas/core/series.py", line 68, in <module>
    from pandas.core import algorithms, base, generic, nanops, ops
  File "/usr/local/lib/python3.7/site-packages/pandas/core/generic.py", line 102, in <module>
    from pandas.io.formats import format as fmt
  File "/usr/local/lib/python3.7/site-packages/pandas/io/formats/format.py", line 71, in <module>
    from pandas.io.common import stringify_path
  File "/usr/local/lib/python3.7/site-packages/pandas/io/common.py", line 7, in <module>
    import mmap
ImportError: /apps/splunk/lib/python3.7/mmap.so: undefined symbol: PL_mmap_page_size

Any Ideas? 

Labels (3)
Tags (1)
0 Karma

andreasz
Path Finder

This is what worked for me: download the Python for Scientific Computing app https://splunkbase.splunk.com/app/2882 and copy the following two files 

/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python3.7/lib-dynload/mmap.cpython-37m-x86_64-linux-gnu.so
/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python3.7/lib-dynload/cmath.cpython-37m-x86_64-linux-gnu.so
 
to /opt/splunk/lib/python3.7
 
Tested it with custom search commands.
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!