topic Re: TA-WebTools in All Apps and Add-ons

TA-WebTools

Kendo213 — Wed, 30 Sep 2020 03:46:56 GMT

I've installed this on a Splunk box running 7.3.1. Initially, the first scheduled CURL input worked, with a simple URI of the following:

https://api.ssllabs.com/api/v3/analyze?host=mydomain.com

I cloned this input, changing the domain in the URI, and the additional input didn't function. I deleted all inputs, and tried again, and still wasn't successful. I've also removed the Webtools app, and re-installed it, which didn't work. I do see these errors in the internal logs, if they're helpful:

2020-01-21 10:09:07,236 ERROR pid=115445 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-webtools/bin/ta_webtools/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-webtools/bin/curl_inputs.py", line 88, in collect_events
input_module.collect_events(self, ew)
File "/opt/splunk/etc/apps/TA-webtools/bin/input_module_curl_inputs.py", line 43, in collect_events
if len(payload)>0:
TypeError: object of type 'NoneType' has no len()

Running curl with SPL does work correctly, and one of the inputs did work at one point. I'm not inserting any headers, or anything special.

Any ideas?

Re: TA-WebTools

efavreau — Tue, 21 Jan 2020 16:53:01 GMT

Probably not your issue, but keep in mind that API has limits and conditions in the terms of service that come with it. If you hit it too much or violate the TOS, it'll block you.

Re: TA-WebTools

Kendo213 — Tue, 21 Jan 2020 16:55:25 GMT

Yeah - that was my first check - but I can do the curl from the server, or with SPL via this add-on from the same device. I'm assuming it isn't that based on testing.

Re: TA-WebTools

jkat54 — Tue, 21 Jan 2020 17:25:18 GMT

The behavior makes me think you're having kvstore issues. What does "index=_internal sourcetype=mongod" say?

Re: TA-WebTools

Kendo213 — Wed, 30 Sep 2020 03:46:59 GMT

Nothing really stands out, but I'm not terribly familiar with this log.

Some common messages from this log on the search head with webtools installed (1 input configured, and I rebooted it around 11:22):

2020-01-21T17:22:55.740Z I ACCESS [conn16] Successfully authenticated as principal __system on local

2020-01-21T17:22:55.724Z I NETWORK [conn16] received client metadata from 127.0.0.1:46302 conn16: { driver: { name: "mongoc", version: "1.13.0" }, os: { type: "Linux", name: "Red Hat Enterprise Linux Server", version: "7.7", architecture: "x86_64" }, platform: "cfg=0x001620c9 posix=200112 stdc=201112 CC=GCC 5.3.0 CFLAGS="-g -fstack-protector-strong -static-libgcc -L/opt/splunk-home/lib/static-libstdc -DPY_BIG..." }

2020-01-21T17:22:55.722Z I NETWORK [listener] connection accepted from 127.0.0.1:46306 #18 (12 connections now open)

2020-01-21T17:22:50.150Z I REPL [rsSync] transition to primary complete; database writes are now permitted

2020-01-21T17:22:47.592Z I STORAGE [initandlisten] ** WARNING: Readahead for /opt/splunk/var/lib/splunk/kvstore/mongo is set to 4096KB

Re: TA-WebTools

Kendo213 — Tue, 21 Jan 2020 17:52:55 GMT

I would add, that I have an input scheduled for every 60 seconds, but in the last 15 minutes have seen no activity in that mongod log.

Re: TA-WebTools

jkat54 — Tue, 21 Jan 2020 17:59:41 GMT

Yeah this log is fun...

To find errors you append this to the search:

| regex "\sE\s"

To find warnings you append this:

|regex "\sW\s"

Re: TA-WebTools

jkat54 — Tue, 21 Jan 2020 18:01:37 GMT

please restart splunk and then look in this log for error messages on startup.

Re: TA-WebTools

Kendo213 — Tue, 21 Jan 2020 18:04:41 GMT

Assuming those regex are correct, I only see 2020-01-21T17:22:47.555Z W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.

I did find the other warning in my previous post by searching for "warning" OR "critical" OR "error"

Re: TA-WebTools

Kendo213 — Tue, 21 Jan 2020 18:10:04 GMT

About 133 events generated during startup - your Error regex returns nothing, and the Warning regex only returns the cipher suite message.

Re: TA-WebTools

jkat54 — Tue, 21 Jan 2020 18:13:15 GMT

Hmm what does the CLI command

 ./splunk show kvstore-status

Say?

Re: TA-WebTools

Kendo213 — Tue, 21 Jan 2020 18:19:32 GMT

This member:
backupRestoreStatus : Ready
date : Tue Jan 21 12:18:27 2020
dateSec : 1579630707.102
disabled : 0
guid : B58F8FD5-DBDB-4FB5-8469-E6F4B7B D68CB
oplogEndTimestamp : Tue Jan 21 12:18:22 2020
oplogEndTimestampSec : 1579630702
oplogStartTimestamp : Fri Oct 4 05:30:05 2019
oplogStartTimestampSec : 1570185005
port : 8191
replicaSet : B58F8FD5-DBDB-4FB5-8469-E6F4B7B D68CB
replicationStatus : KV store captain
standalone : 1
status : ready

KV store members:
127.0.0.1:8191
configVersion : 1
electionDate : Tue Jan 21 12:08:10 2020
electionDateSec : 1579630090
hostAndPort : 127.0.0.1:8191
optimeDate : Tue Jan 21 12:18:22 2020
optimeDateSec : 1579630702
replicationStatus : KV store captain
uptime : 617

Re: TA-WebTools

Kendo213 — Tue, 21 Jan 2020 18:24:18 GMT

One thing to mention, is that I had a working input, cloned it, and accidentally left the URI as the same value...but with a different input name. I assume this wouldn't cause issues, just noting it.

Re: TA-WebTools

jkat54 — Tue, 21 Jan 2020 18:52:50 GMT

If I recall correctly, it appends the NAME of the input to the kvstore collection name so that you get a different kvstore collection for each input.

Re: TA-WebTools

Kendo213 — Tue, 21 Jan 2020 18:54:59 GMT

Interesting. Yeah, I'm at a loss. It worked for that first input, and then never again. I can't wrap my head around how that's possible. And it's specific to the scheduled curl inputs, the command itself works fine.

Re: TA-WebTools

jkat54 — Tue, 21 Jan 2020 18:57:39 GMT

Just seeing this error:

if len(payload)>0:
TypeError: object of type 'NoneType' has no len()

This happens when the API returns 0 results. Something is wrong with the query to the api. It's probably the parameter you're passing that is causing issues with my code for the scripted inputs.

Can you use the | curl command and send it to. | collect index=indexName marker=curlInputName?

Or send it to outputlookup, etc?

Then schedule an alert/report that automates it?

Re: TA-WebTools

Kendo213 — Wed, 30 Sep 2020 03:47:05 GMT

It returns data every time - whether it's a valid domain status, or the fact that they're too busy (either limiting me, or everyone). I can output it to a CSV successfully, and use inputlookup to view it. I get a curl_response_url, curl_message, and curl_status.

The issue is that I need this ingested so I can leverage it in ITSI as an example. I suppose I could do it this way, but that would require me to put the application on the ITSI box which I'm trying to avoid.

The only parameter I'm passing is the URI, I'm not inserting headers/etc. I'm confused on how it worked the first time with the same URI 😕

Re: TA-WebTools

Kendo213 — Tue, 21 Jan 2020 19:20:49 GMT

I've also simply tried passing https://api.ssllabs.com/api/v3/ with no parameters. I would expect a curl response 404, a message (html) and the response URL. At least that's what I see if I |curl method=get uri=https://api.ssllabs.com/api/v3/.

Re: TA-WebTools

Kendo213 — Tue, 21 Jan 2020 19:46:11 GMT

And I did go back to the results that did come in, and enter the same URI that showed in the Splunk logs. So I don't believe this can be the result of any parameters I'm passing - I'm taking parameters that are in the fields in the logs that did make it in.

Re: TA-WebTools

Kendo213 — Tue, 21 Jan 2020 20:02:33 GMT

I did try the |collect route, and that does work. The only issue is that (of course) it isn't a JSON payload, so it's something I would have to rex out. I think this can work as a workaround -- I do think it might be worth investigating why this stopped working despite no changes on my end other than deleting/cloning/adding inputs.