topic Re: Numpi import through Splunk script in Getting Data In

Numpi import through Splunk script

hkmurali — Tue, 29 Sep 2020 12:35:06 GMT

I'm trying to parse a log file and written a python script to parse it
However when I run it in Splunk search app, only the first line is being executed
I had asked this question before , but now I realize it's because of import numpy command
Is there anything special I should do for using numpy commands in a python script, placed in the bin folder

print("Test Message for log")
import numpy
lines = [] # Declare an empty list named "lines"
i=0
end_position=[]
with open ('$SPLUNK_HOME\etc\apps\search\bin\prm_rca.20161129_0000.log', 'rt') as in_file:
for line in in_file: # For each line of text in in_file, where the data is named "line",
print("Test Message within a loop")
lines.append(line.rstrip('\n')) # add that line to our list of lines, stripping newlines.
if "-----------------------------------------------------------------" in line:
end_position.append(i)
i=i+1
Please help
Objective : parse a very complex log file and make some meaningful analysis

Re: Numpi import through Splunk script

woodcock — Tue, 24 Jan 2017 16:42:25 GMT

I do not understand your situation at all. Show us your configuration files and a sample of your broken results and a mockup of your expected results.

Re: Numpi import through Splunk script

hkmurali — Fri, 27 Jan 2017 06:50:47 GMT

Hi
The basic objective is to do the right kind of parsing.
The structure of the log file is in this format :

                Top 20 CPU Consuming Processes               
----------------------------------------------------------------- USER PID %CPU %MEM VSZ RSS
TTY STAT START TIME COMMAND
oracle 18747 8.4 4.5 6545080
1480668 ? Ss Nov28 5:03

oracleprod (LOCAL=NO) etc...
                Top 20 Memory Consuming Processes            
----------------------------------------------------------------- USER PID %CPU %MEM VSZ RSS
TTY STAT START TIME COMMAND
oracle 13342 6.0 10.9 6551532
3587824 ? Ss Nov28 53:39

oracleprod (LOCAL=NO) etc..
                   Server Processes                          

and so forth.. so there are 8 such tables in a single log file. Basic splunk parsing doesnt help and its not splitting into separate tables . So we thought we'll write a python script to parse the log files in splunk. We put the python script in the bin folder and under scripts option, selected the .py file , went to search app and ran the file. It's only generating the first line of code which is a test print command, that I wanted to check if its reading correctly or not. After that its not reading any of the lines.

My question is : the python sript written uses a Numpy library and hence wanted to check if this is causing the issue for Splunk ,unable to import numpy library to read the python file

I can send u the code and log file, if you can provide me ur mail id

Re: Numpi import through Splunk script

jangid — Thu, 15 Feb 2018 10:02:20 GMT

Hi @hkmurali

By default Splunk doesn't ship numpy package - I guess your script is not able to find the numpy package, have you installed numpy package inside Splunk?

try to install Python for Scientific Computing app and check whether this will resolve your issue or not.

Namaste,
Manoj