Splunk python with pandas

[to4kawa]

Splunk python3 can't work import pandas
How should I do?

sample:

% $SPLUNK_HOME/bin/splunk cmd python3
>>> import pandas
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'

and how does custom apps use python3?
where should I put on python.version=python3?

[stevejonn]

In most cases this error in Python generally raised:

• You haven't installed Pandas explicitly with pip install pandas.
• You may have different Python versions on your computer and Pandas is not installed for the particular version you're using.

You can run the following command in your Linux/MacOS/Windows terminal.

pip install pandas

To be sure you are not having multiple Python versions that are confusing, you should run following commands:

python3 -m pip install pandas
python3 -c 'import pandas'

[seva98 · Accepted solution]

Try manually downloading pandas and copy them to Splunk\etc\apps\<APP>\bin folder then in script that you have in that bin folder, for example Splunk\etc\apps\<APP>\bin\main.py add following:

import sys
sys.path.append('./pandas')
sys.path.insert(0, './pandas')

It is lazy workaround but importing should now work.

[sarvesh_11]

Hello,

I tried this work around for pandas.

but i am getting the below error while calling pandas.

Traceback (most recent call last):
File "sample.py", line 38, in <module>
df = pandas.DataFrame(data)
NameError: name 'pandas' is not defined

while i added syspath append and insert, i removed import pandas from the python file.

[to4kawa]

@seva98
thank you. I can't find this solution.

[to4kawa]

https://docs.splunk.com/Documentation/Splunk/8.0.3/AdvancedDev/ScriptWriting

[to4kawa]

https://github.com/splunk/mltk-algo-contrib

[to4kawa]

https://docs.splunk.com/Documentation/MLApp/latest/API/AddPython3Libs

[to4kawa]

https://static.rainfocus.com/splunk/splunkconf18/sess/1523381218451001Aj1y/finalPDF/FN1409_GitHubCol...

[to4kawa]

https://answers.splunk.com/answers/686808/splunk-machine-learning-toolkit-install-additional.html

[to4kawa]

https://answers.splunk.com/answers/795799/how-to-import-util-when-using-exec-anaconda.html

[PavelP]

Hello @to4kawa,

I think the pandas module wasn't properly installed. Can you list your steps and reference to the documentation you've used to install pandas python module?

[to4kawa]

hi @PaveIP

!#/usr/bin/env python

It run with python in my OS(e.g. Anaconda) . I can use pandas ,also.

I'm trying to publish Apps, and considering windows, if pandas can be used with Splunk's standard python, I'm trying to change to that method like following:

!#$SPLUNK_HOME/bin/python3

However, I have not found it yet. Should I copy Python for Scientific Computing/exec_anaconda.py ?

reference: https://github.com/splunk/Splunk-python-for-scientific-computing/tree/master/package

[PavelP]

I'm not sure, that can work

[to4kawa]

Thanks @PaveIP
No problem.

[to4kawa]

https://answers.splunk.com/answers/681860/python-for-scientific-computing-app-error-when-imp.html

[to4kawa]

https://stackoverflow.com/questions/55275491/splunk-splunk-python-sdk-how-to-include-pandas-numpy-to...

[to4kawa]

https://dev.splunk.com/enterprise/docs/developapps/customsearchcommands/createcustomsearchcmd/

[to4kawa]

reference:https://docs.splunk.com/Documentation/Splunk/latest/Python3Migration/ChangesEnterprise#Python_interp...

[to4kawa]

sample usage:

transforms.conf

[freqserver]
external_cmd = freq.py domain
external_type = python
python.version = python2
fields_list = domain, frequency

ref: https://answers.splunk.com/answering/816676/view.html