All Apps and Add-ons

TA-WebTools

Communicator

I've installed this on a Splunk box running 7.3.1. Initially, the first scheduled CURL input worked, with a simple URI of the following:

https://api.ssllabs.com/api/v3/analyze?host=mydomain.com

I cloned this input, changing the domain in the URI, and the additional input didn't function. I deleted all inputs, and tried again, and still wasn't successful. I've also removed the Webtools app, and re-installed it, which didn't work. I do see these errors in the internal logs, if they're helpful:

2020-01-21 10:09:07,236 ERROR pid=115445 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-webtools/bin/ta_webtools/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-webtools/bin/curl_inputs.py", line 88, in collect_events
input_module.collect_events(self, ew)
File "/opt/splunk/etc/apps/TA-webtools/bin/input_module_curl_inputs.py", line 43, in collect_events
if len(payload)>0:
TypeError: object of type 'NoneType' has no len()

Running curl with SPL does work correctly, and one of the inputs did work at one point. I'm not inserting any headers, or anything special.

Any ideas?

0 Karma

SplunkTrust
SplunkTrust

Just seeing this error:

if len(payload)>0:
TypeError: object of type 'NoneType' has no len()

This happens when the API returns 0 results. Something is wrong with the query to the api. It's probably the parameter you're passing that is causing issues with my code for the scripted inputs.

Can you use the | curl command and send it to. | collect index=indexName marker=curlInputName?

Or send it to outputlookup, etc?

Then schedule an alert/report that automates it?

0 Karma

Communicator

I did try the |collect route, and that does work. The only issue is that (of course) it isn't a JSON payload, so it's something I would have to rex out. I think this can work as a workaround -- I do think it might be worth investigating why this stopped working despite no changes on my end other than deleting/cloning/adding inputs.

0 Karma

Communicator

It returns data every time - whether it's a valid domain status, or the fact that they're too busy (either limiting me, or everyone). I can output it to a CSV successfully, and use inputlookup to view it. I get a curl_response_url, curl_message, and curl_status.

The issue is that I need this ingested so I can leverage it in ITSI as an example. I suppose I could do it this way, but that would require me to put the application on the ITSI box which I'm trying to avoid.

The only parameter I'm passing is the URI, I'm not inserting headers/etc. I'm confused on how it worked the first time with the same URI 😕

0 Karma

Communicator

I've also simply tried passing https://api.ssllabs.com/api/v3/ with no parameters. I would expect a curl response 404, a message (html) and the response URL. At least that's what I see if I |curl method=get uri=https://api.ssllabs.com/api/v3/.

0 Karma

Communicator

And I did go back to the results that did come in, and enter the same URI that showed in the Splunk logs. So I don't believe this can be the result of any parameters I'm passing - I'm taking parameters that are in the fields in the logs that did make it in.

0 Karma

SplunkTrust
SplunkTrust

The behavior makes me think you're having kvstore issues. What does "index=_internal sourcetype=mongod" say?

0 Karma

Communicator

So, this is working now after doing the following per input-

1) I configured it as I've documented here, with the URI and nothing else
2) Verified no data was coming in
3) Switched the URI on the input to leverage a test API (https://docs.postman-echo.com/?version=latest)
4) Verified data was coming in
5) Switched the URI back to the original URI (ssllabs) and it began ingesting data successfully

Once it's working, I actually see the GET request in the internal logs - on the failing inputs I don't see this. As an example, 2020-01-22 11:12:52,278 DEBUG pid=88838 tid=MainThread file=connectionpool.py:_make_request:400 | https://api.ssllabs.com:443 "GET /api/v3/analyze?host=mydomain.com HTTP/1.1" 200 None

Any ideas on why this would happen? It seems like once data is ingested, I can then use that specific input and just tweak the URI.

0 Karma

Communicator

Nothing really stands out, but I'm not terribly familiar with this log.

Some common messages from this log on the search head with webtools installed (1 input configured, and I rebooted it around 11:22):

2020-01-21T17:22:55.740Z I ACCESS [conn16] Successfully authenticated as principal __system on local

2020-01-21T17:22:55.724Z I NETWORK [conn16] received client metadata from 127.0.0.1:46302 conn16: { driver: { name: "mongoc", version: "1.13.0" }, os: { type: "Linux", name: "Red Hat Enterprise Linux Server", version: "7.7", architecture: "x86_64" }, platform: "cfg=0x001620c9 posix=200112 stdc=201112 CC=GCC 5.3.0 CFLAGS="-g -fstack-protector-strong -static-libgcc -L/opt/splunk-home/lib/static-libstdc -DPY_BIG..." }

2020-01-21T17:22:55.722Z I NETWORK [listener] connection accepted from 127.0.0.1:46306 #18 (12 connections now open)

2020-01-21T17:22:50.150Z I REPL [rsSync] transition to primary complete; database writes are now permitted

2020-01-21T17:22:47.592Z I STORAGE [initandlisten] ** WARNING: Readahead for /opt/splunk/var/lib/splunk/kvstore/mongo is set to 4096KB

0 Karma

Communicator

I would add, that I have an input scheduled for every 60 seconds, but in the last 15 minutes have seen no activity in that mongod log.

0 Karma

SplunkTrust
SplunkTrust

Yeah this log is fun...

To find errors you append this to the search:

| regex "\sE\s"

To find warnings you append this:

|regex "\sW\s"
0 Karma

Communicator

Assuming those regex are correct, I only see 2020-01-21T17:22:47.555Z W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.

I did find the other warning in my previous post by searching for "warning" OR "critical" OR "error"

0 Karma

SplunkTrust
SplunkTrust

Hmm what does the CLI command

 ./splunk show kvstore-status 

Say?

0 Karma

Communicator

This member:
backupRestoreStatus : Ready
date : Tue Jan 21 12:18:27 2020
dateSec : 1579630707.102
disabled : 0
guid : B58F8FD5-DBDB-4FB5-8469-E6F4B7B D68CB
oplogEndTimestamp : Tue Jan 21 12:18:22 2020
oplogEndTimestampSec : 1579630702
oplogStartTimestamp : Fri Oct 4 05:30:05 2019
oplogStartTimestampSec : 1570185005
port : 8191
replicaSet : B58F8FD5-DBDB-4FB5-8469-E6F4B7B D68CB
replicationStatus : KV store captain
standalone : 1
status : ready

KV store members:
127.0.0.1:8191
configVersion : 1
electionDate : Tue Jan 21 12:08:10 2020
electionDateSec : 1579630090
hostAndPort : 127.0.0.1:8191
optimeDate : Tue Jan 21 12:18:22 2020
optimeDateSec : 1579630702
replicationStatus : KV store captain
uptime : 617

0 Karma

Communicator

One thing to mention, is that I had a working input, cloned it, and accidentally left the URI as the same value...but with a different input name. I assume this wouldn't cause issues, just noting it.

0 Karma

SplunkTrust
SplunkTrust

If I recall correctly, it appends the NAME of the input to the kvstore collection name so that you get a different kvstore collection for each input.

0 Karma

Communicator

Interesting. Yeah, I'm at a loss. It worked for that first input, and then never again. I can't wrap my head around how that's possible. And it's specific to the scheduled curl inputs, the command itself works fine.

0 Karma

SplunkTrust
SplunkTrust

please restart splunk and then look in this log for error messages on startup.

0 Karma

Communicator

About 133 events generated during startup - your Error regex returns nothing, and the Warning regex only returns the cipher suite message.

0 Karma

Builder

Probably not your issue, but keep in mind that API has limits and conditions in the terms of service that come with it. If you hit it too much or violate the TOS, it'll block you.

###

If this reply helps you, an upvote would be appreciated.
0 Karma

Communicator

Yeah - that was my first check - but I can do the curl from the server, or with SPL via this add-on from the same device. I'm assuming it isn't that based on testing.

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!