TA-WebTools

[Kendo213]

I've installed this on a Splunk box running 7.3.1. Initially, the first scheduled CURL input worked, with a simple URI of the following:

https://api.ssllabs.com/api/v3/analyze?host=mydomain.com

I cloned this input, changing the domain in the URI, and the additional input didn't function. I deleted all inputs, and tried again, and still wasn't successful. I've also removed the Webtools app, and re-installed it, which didn't work. I do see these errors in the internal logs, if they're helpful:

2020-01-21 10:09:07,236 ERROR pid=115445 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-webtools/bin/ta_webtools/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-webtools/bin/curl_inputs.py", line 88, in collect_events
input_module.collect_events(self, ew)
File "/opt/splunk/etc/apps/TA-webtools/bin/input_module_curl_inputs.py", line 43, in collect_events
if len(payload)>0:
TypeError: object of type 'NoneType' has no len()

Running curl with SPL does work correctly, and one of the inputs did work at one point. I'm not inserting any headers, or anything special.

Any ideas?

[jjw420]

I fixed this issue and raised a PR on the owners github repo.   

See my fix here:
https://github.com/bentleymi/ta-webtools/pull/16

No "none" checks are performed which breaks existing configuration.   My fix (which we are running in Production) does these checks.

[jkat54]

Just seeing this error:

if len(payload)>0:
TypeError: object of type 'NoneType' has no len()

This happens when the API returns 0 results. Something is wrong with the query to the api. It's probably the parameter you're passing that is causing issues with my code for the scripted inputs.

Can you use the | curl command and send it to. | collect index=indexName marker=curlInputName?

Or send it to outputlookup, etc?

Then schedule an alert/report that automates it?

[Kendo213]

I did try the |collect route, and that does work. The only issue is that (of course) it isn't a JSON payload, so it's something I would have to rex out. I think this can work as a workaround -- I do think it might be worth investigating why this stopped working despite no changes on my end other than deleting/cloning/adding inputs.

[Kendo213]

It returns data every time - whether it's a valid domain status, or the fact that they're too busy (either limiting me, or everyone). I can output it to a CSV successfully, and use inputlookup to view it. I get a curl_response_url, curl_message, and curl_status.

The issue is that I need this ingested so I can leverage it in ITSI as an example. I suppose I could do it this way, but that would require me to put the application on the ITSI box which I'm trying to avoid.

The only parameter I'm passing is the URI, I'm not inserting headers/etc. I'm confused on how it worked the first time with the same URI 😕

[Kendo213]

I've also simply tried passing https://api.ssllabs.com/api/v3/ with no parameters. I would expect a curl response 404, a message (html) and the response URL. At least that's what I see if I |curl method=get uri=https://api.ssllabs.com/api/v3/.

[Kendo213]

And I did go back to the results that did come in, and enter the same URI that showed in the Splunk logs. So I don't believe this can be the result of any parameters I'm passing - I'm taking parameters that are in the fields in the logs that did make it in.

[jkat54]

The behavior makes me think you're having kvstore issues. What does "index=_internal sourcetype=mongod" say?

[Kendo213]

So, this is working now after doing the following per input-

1) I configured it as I've documented here, with the URI and nothing else
2) Verified no data was coming in
3) Switched the URI on the input to leverage a test API (https://docs.postman-echo.com/?version=latest)
4) Verified data was coming in
5) Switched the URI back to the original URI (ssllabs) and it began ingesting data successfully

Once it's working, I actually see the GET request in the internal logs - on the failing inputs I don't see this. As an example, 2020-01-22 11:12:52,278 DEBUG pid=88838 tid=MainThread file=connectionpool.py:_make_request:400 | https://api.ssllabs.com:443 "GET /api/v3/analyze?host=mydomain.com HTTP/1.1" 200 None

Any ideas on why this would happen? It seems like once data is ingested, I can then use that specific input and just tweak the URI.

[Kendo213]

Nothing really stands out, but I'm not terribly familiar with this log.

Some common messages from this log on the search head with webtools installed (1 input configured, and I rebooted it around 11:22):

2020-01-21T17:22:55.740Z I ACCESS [conn16] Successfully authenticated as principal __system on local

2020-01-21T17:22:55.724Z I NETWORK [conn16] received client metadata from 127.0.0.1:46302 conn16: { driver: { name: "mongoc", version: "1.13.0" }, os: { type: "Linux", name: "Red Hat Enterprise Linux Server", version: "7.7", architecture: "x86_64" }, platform: "cfg=0x001620c9 posix=200112 stdc=201112 CC=GCC 5.3.0 CFLAGS="-g -fstack-protector-strong -static-libgcc -L/opt/splunk-home/lib/static-libstdc -DPY_BIG..." }

2020-01-21T17:22:55.722Z I NETWORK [listener] connection accepted from 127.0.0.1:46306 #18 (12 connections now open)

2020-01-21T17:22:50.150Z I REPL [rsSync] transition to primary complete; database writes are now permitted

2020-01-21T17:22:47.592Z I STORAGE [initandlisten] ** WARNING: Readahead for /opt/splunk/var/lib/splunk/kvstore/mongo is set to 4096KB

[Kendo213]

I would add, that I have an input scheduled for every 60 seconds, but in the last 15 minutes have seen no activity in that mongod log.

[jkat54]

Yeah this log is fun...

To find errors you append this to the search:

| regex "\sE\s"

To find warnings you append this:

|regex "\sW\s"

[Kendo213]

Assuming those regex are correct, I only see 2020-01-21T17:22:47.555Z W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.

I did find the other warning in my previous post by searching for "warning" OR "critical" OR "error"

[jkat54]

Hmm what does the CLI command

 ./splunk show kvstore-status 

Say?

[Kendo213]

This member:
backupRestoreStatus : Ready
date : Tue Jan 21 12:18:27 2020
dateSec : 1579630707.102
disabled : 0
guid : B58F8FD5-DBDB-4FB5-8469-E6F4B7B D68CB
oplogEndTimestamp : Tue Jan 21 12:18:22 2020
oplogEndTimestampSec : 1579630702
oplogStartTimestamp : Fri Oct 4 05:30:05 2019
oplogStartTimestampSec : 1570185005
port : 8191
replicaSet : B58F8FD5-DBDB-4FB5-8469-E6F4B7B D68CB
replicationStatus : KV store captain
standalone : 1
status : ready

KV store members:
127.0.0.1:8191
configVersion : 1
electionDate : Tue Jan 21 12:08:10 2020
electionDateSec : 1579630090
hostAndPort : 127.0.0.1:8191
optimeDate : Tue Jan 21 12:18:22 2020
optimeDateSec : 1579630702
replicationStatus : KV store captain
uptime : 617

[Kendo213]

One thing to mention, is that I had a working input, cloned it, and accidentally left the URI as the same value...but with a different input name. I assume this wouldn't cause issues, just noting it.

[jkat54]

If I recall correctly, it appends the NAME of the input to the kvstore collection name so that you get a different kvstore collection for each input.

[Kendo213]

Interesting. Yeah, I'm at a loss. It worked for that first input, and then never again. I can't wrap my head around how that's possible. And it's specific to the scheduled curl inputs, the command itself works fine.

[jkat54]

please restart splunk and then look in this log for error messages on startup.

[Kendo213]

About 133 events generated during startup - your Error regex returns nothing, and the Warning regex only returns the cipher suite message.

[efavreau]

Probably not your issue, but keep in mind that API has limits and conditions in the terms of service that come with it. If you hit it too much or violate the TOS, it'll block you.