About blangrill

blangrill · ‎01-07-2019

I have Splunk configured to pull data from a Nessus Professional instance and for the most part it works fine but I noticed several recent scan reports weren't showing up. After some investigating, there were errors like the following during each pull attempt (every 12 hours): 2019-01-07 07:44:06,351 ERROR pid=9520 tid=MainThread file=nessus_rest_client.py:request:100 | Failed to connect https://<splunk server>/scans/166, reason=Traceback (most recent call last): File "E:\splunk\etc\apps\Splunk_TA_nessus\bin\nessus_rest_client.py", line 80, in request headers=headers) File "E:\splunk\etc\apps\Splunk_TA_nessus\bin\splunktalib\httplib2\__init__.py", line 1593, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "E:\splunk\etc\apps\Splunk_TA_nessus\bin\splunktalib\httplib2\__init__.py", line 1335, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "E:\splunk\etc\apps\Splunk_TA_nessus\bin\splunktalib\httplib2\__init__.py", line 1291, in _conn_request response = conn.getresponse() File "E:\splunk\Python-2.7\Lib\httplib.py", line 1121, in getresponse response.begin() File "E:\splunk\Python-2.7\Lib\httplib.py", line 438, in begin version, status, reason = self._read_status() File "E:\splunk\Python-2.7\Lib\httplib.py", line 394, in _read_status line = self.fp.readline(_MAXLINE + 1) File "E:\splunk\Python-2.7\Lib\socket.py", line 480, in readline data = self._sock.recv(self._rbufsize) File "E:\splunk\Python-2.7\Lib\ssl.py", line 772, in recv return self.read(buflen) File "E:\splunk\Python-2.7\Lib\ssl.py", line 659, in read v = self._sslobj.read(len) SSLError: ('The read operation timed out',) I noted that over the past few weeks the same scan numbers were showing up repeatedly and checking in Nessus, these are indeed the missing scan reports. These happen to be the largest scan results so I suspect the time out is just that the Nessus server isn't responding fast enough and the Splunk app closes the connection. Is there a way to increase the read timeout for this situation?

blangrill · ‎10-10-2018

The answer is quite simple. Adding -E to the Python3 call tells it to ignore other PYTHONPATH variables and it is able to run successfully. The batch file I am calling as a Splunk scripted input now looks like: @ECHO off C:\Python37\python.exe -E "<path to python3 script>"

blangrill · ‎10-02-2018

Splunk still uses Python 2.7 internally but has the ability to call external scripts to generate data via Scripted Inputs I would like to pull data using an external API which has a Python 3 library. I have installed Python 3 to a separate place on the file system and written a Windows batch script, which calls this and invokes the Python 3 API. When run from the OS, this generates the data I would like, but if I try to add this batch script as an input to Splunk, I get an error along the lines of the following: ERROR ExecProcessor - message from "e:\splunk\bin\scripts\test.bat" Fatal Python error: Py_Initialize: unable to load the file system codec ERROR ExecProcessor - message from "e:\splunk\bin\scripts\test.bat" File "E:\splunk\Python-2.7\Lib\encodings\__init__.py", line 123 ERROR ExecProcessor - message from "e:\splunk\bin\scripts\test.bat" raise CodecRegistryError,\ ERROR ExecProcessor - message from "e:\splunk\bin\scripts\test.bat" ^ ERROR ExecProcessor - message from "e:\splunk\bin\scripts\test.bat" SyntaxError: invalid syntax I get a similar error even if I minimally set the batch script to only contain a minimal statement. <path to python3>\python.exe -c 'print("Hello")' This implies the problem is something from Python 2 being passed to the Python 3 environment since the Splunk script call is ultimately from that. I asked Splunk support about this and was told it "isn't supported" and was directed to contact professional services which seems like overkill for what is likely just an environment issue. Is there a way to wrap the python commands such that this still works? As of now my workaround is to have the batch file called as a scheduled task in Windows and write the results to a file which is then monitored by Splunk. Additionally, at some point Splunk will make the jump internally to Python 3 and likely people will have the reverse problem with older Python 2.X libraries.

blangrill · ‎06-29-2018

This was the trick. Thank you!

blangrill · ‎06-27-2018

I definitely appreciate the effort. If there are upcoming changes to the way data is pulled this probably doesn't matter too much. I tried escaping the pipe and the quotes and always got the same error. Yes that is it in full above from the splunk GUI. Here is the lines directly from the oms_inputs.log file: 2018-06-27 16:46:12,069 ERROR pid=4584 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events. Traceback (most recent call last): File "E:\splunk\etc\apps\TA-OMS_Inputs\bin\ta_oms_inputs\modinput_wrapper\base_modinput.py", line 127, in stream_events self.collect_events(ew) File "E:\splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py", line 96, in collect_events input_module.collect_events(self, ew) File "E:\splunk\etc\apps\TA-OMS_Inputs\bin\input_module_oms_inputs.py", line 86, in collect_events search_id = data["id"].split("/") KeyError: 'id' Any idea if there is a way to see the query as submitted to OMS using the correlation_id? My OMS guy wasn't aware of one.

blangrill · ‎06-27-2018

Use case: I want to pull a specific set of security events from OMS into Splunk. Within OMS log search, querying for: SecurityBaseline gives me all events from the set, and filtering for the events I want is: SecurityBaseline | AnalyzeResult=="Failed" No issues. In Splunk OMS inputs, setting my "OMS Query" to the first one does indeed work and start pulling events from that set (it is huge). However, if I change it to the latter with the filter no events are pulled and the following error show up in the oms app log: tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events. Traceback (most recent call last): File "E:\splunk\etc\apps\TA-OMS_Inputs\bin\ta_oms_inputs\modinput_wrapper\base_modinput.py", line 127, in stream_events self.collect_events(ew) File "E:\splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py", line 96, in collect_events input_module.collect_events(self, ew) File "E:\splunk\etc\apps\TA-OMS_Inputs\bin\input_module_oms_inputs.py", line 86, in collect_events search_id = data["id"].split("/") Relatively new to OMS, but I think this is probably just a syntax issue somewhere...any place for reference queries or different way I should do this? (If it matters, I am using OMS Inputs app v 1.3.3 on Splunk Enterprise 7.0.2)

Posts	6
Solutions	1
Karma Given	1
Karma Received	2
Member Since	‎03-02-2018

Online Status	Offline
Date Last Visited	‎06-05-2020 02:04 AM

Increase Timeout for Querying Nessus Scan Data

In Splunk Enterprise, can you help me call an exte...

Issues with OMS Query Filters

Increase Timeout for Querying Nessus Scan Data

Re: In Splunk Enterprise, can you help me call an ...

In Splunk Enterprise, can you help me call an exte...

Re: Issues with OMS Query Filters

Re: Issues with OMS Query Filters

Issues with OMS Query Filters