All Apps and Add-ons

Website Input & case sensitivity issue: Is it possible to configure "authorization: command" to be "Authorization"?

cmodyssey
Explorer

Hi,

My workaround reply is still waiting for the moderator to accept the posting.

In the mean-time, I have a new problem that's more difficult to get around.

The device that I am getting Website input to scrape is a BBQ temperature controller, so I don't have much control over what it expects or receives.

The problem I have now come across has to do with HTTP authentication.

With some Wireshark, wget, and telnet testing (manually sending the HTTP request messages), I have figured out what the BBQ temperature controller does not like about HTTP authentication from Website input.

It's the "authorization: command". Website input has the first character in lower-case, which my controller does not like. Wget has the first character in upper-case, which my controller does like and works.

Please would it be possible to have it so that the authorization command can be configurable to be "authorization" or "Authorization" (unless it's really meant to be "Authorization").

Thanks,

Richard.

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

Ok, so they use python27 httplib2 with the webinput app & splunk core libraries. This is from httplib2 code and authorization is lowercase.

class BasicAuthentication(Authentication):
    def __init__(self, credentials, host, request_uri, headers, response, content, http):
        Authentication.__init__(self, credentials, host, request_uri, headers, response, content, http)

    def request(self, method, request_uri, headers, content):
        """Modify the request headers to add the appropriate
        Authorization header."""
        headers['authorization'] = 'Basic ' + base64.b64encode("%s:%s" % self.credentials).strip()

https://github.com/jcgregorio/httplib2/blob/master/python2/httplib2/__init__.py#L507

If you want to hack a fix... just search your server for httplib2's __init__.py and edit line 507.

Or edit line 270 on web_input.py in the web input app's bin directory. Add a new header:

if user_agent is not None:
 logger.info("Setting user-agent=%s", user_agent)
 headers['User-Agent'] = user_agent
 domain = ""
 cred = [username,password,domain]
 headers['Authorization'] = 'Basic ' + base64.b64encode("%s:%s" % cred).strip()

username, & password variables already exist at this point and so we just wrap them up into an array called cred with a blank domain name and then strip and base64encode the array the same way http lib2 does it... only now we set the header named Authorization instead of authorization. If the issue persists, the next step may be to remove the old 'authorization' header using something like headers['authorizaton'].remove() .

View solution in original post

jkat54
SplunkTrust
SplunkTrust

Ok, so they use python27 httplib2 with the webinput app & splunk core libraries. This is from httplib2 code and authorization is lowercase.

class BasicAuthentication(Authentication):
    def __init__(self, credentials, host, request_uri, headers, response, content, http):
        Authentication.__init__(self, credentials, host, request_uri, headers, response, content, http)

    def request(self, method, request_uri, headers, content):
        """Modify the request headers to add the appropriate
        Authorization header."""
        headers['authorization'] = 'Basic ' + base64.b64encode("%s:%s" % self.credentials).strip()

https://github.com/jcgregorio/httplib2/blob/master/python2/httplib2/__init__.py#L507

If you want to hack a fix... just search your server for httplib2's __init__.py and edit line 507.

Or edit line 270 on web_input.py in the web input app's bin directory. Add a new header:

if user_agent is not None:
 logger.info("Setting user-agent=%s", user_agent)
 headers['User-Agent'] = user_agent
 domain = ""
 cred = [username,password,domain]
 headers['Authorization'] = 'Basic ' + base64.b64encode("%s:%s" % cred).strip()

username, & password variables already exist at this point and so we just wrap them up into an array called cred with a blank domain name and then strip and base64encode the array the same way http lib2 does it... only now we set the header named Authorization instead of authorization. If the issue persists, the next step may be to remove the old 'authorization' header using something like headers['authorizaton'].remove() .

cmodyssey
Explorer

Hi,

Thanks for those suggestions.

Putting the user and pass in the URL stopped Website Input from working altogether.

I edited file /opt/splunk/lib/python2.7/site-packages/httplib2/init.py to change class BasicAuthentication to send "Authorization" instead of "authorization" did the trick!

Now to the fun of graphing the following events:

    response_size="1235" encoding="ascii" request_time="566.810131073" match="695" match="3560" match="OPEN" match="1800" match="100" raw_match_count="5" response_code="200"
    response_size="1235" encoding="ascii" request_time="569.531202316" match="695" match="3560" match="OPEN" match="1800" match="100" raw_match_count="5" response_code="200"
    response_size="1235" encoding="ascii" request_time="576.872110367" match="699" match="3560" match="OPEN" match="1800" match="100" raw_match_count="5" response_code="200"
    response_size="1235" encoding="ascii" request_time="565.46998024" match="697" match="3560" match="OPEN" match="1800" match="100" raw_match_count="5" response_code="200"
    response_size="1235" encoding="ascii" request_time="564.378976822" match="695" match="3560" match="OPEN" match="1800" match="100" raw_match_count="5" response_code="200"
    response_size="1235" encoding="ascii" request_time="568.603992462" match="695" match="3560" match="OPEN" match="1800" match="100" raw_match_count="5" response_code="200"
    response_size="1235" encoding="ascii" request_time="600.687980652" match="695" match="3560" match="OPEN" match="1800" match="100" raw_match_count="5" response_code="200"
    response_size="1235" encoding="ascii" request_time="554.352998734" match="695" match="3560" match="OPEN" match="1800" match="100" raw_match_count="5" response_code="200"
    response_size="1235" encoding="ascii" request_time="569.879055023" match="695" match="3560" match="OPEN" match="1800" match="100" raw_match_count="5" response_code="200"
    response_size="1235" encoding="ascii" request_time="574.70703125" match="695" match="3560" match="OPEN" match="1800" match="100" raw_match_count="5" response_code="200"
    response_size="1235" encoding="ascii" request_time="568.139076233" match="695" match="3560" match="OPEN" match="1800" match="100" raw_match_count="5" response_code="200"
    response_size="1235" encoding="ascii" request_time="577.039003372" match="699" match="3560" match="OPEN" match="1800" match="100" raw_match_count="5" response_code="200"
    response_size="1235" encoding="ascii" request_time="573.582172394" match="697" match="3560" match="OPEN" match="1800" match="100" raw_match_count="5" response_code="200"
    response_size="1235" encoding="ascii" request_time="579.028129578" match="697" match="3560" match="OPEN" match="1800" match="100" raw_match_count="5" response_code="200"

Thanks very much for your help 🙂

0 Karma

jkat54
SplunkTrust
SplunkTrust

I converted the comment to an answer. Can you mark it as the answer?

Glad I could help, not sure you should modify httplib2 because those folks look like they know a thing or two about http requests ;-). You might try forcing the new header instead because if you have other apps built on pythons httplib2 they'll get slightly different than expected results (possibly). Or maybe you could care less!

0 Karma

cmodyssey
Explorer

Hi,

Sure, I've just marked your answer 🙂

I know what you mean about leaving the solution as modifying httplib2, as if I was to choose who was right, Python or my BBQ controller, I would have thought that Python would be right.

I'm going to stick with that as my solution though, the good thing is that Python is specific to Splunk and I won't be doing much else with my Splunk.

If I get other authentication problems, I know how to determine the difference between wget and Splunk and now, thanks to you, know where the relevant http functionality is coded.

Thanks again,

Richard.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Many thanks to you as well for accepting the answer and the up-votes. I really appreciate you coming back with updates so quickly.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Please give us examples of the command that works vs the one that doesnt?

I'm confused when you say its the "authorization: command".

For example:

CURL command looks like this:
curl...

Wireshark shows command coming from app as this:
curl ...

0 Karma

cmodyssey
Explorer

Hi,

Here is the HTTP request that comes from Website input:

    GET /all.xml HTTP/1.1
    Host: 192.168.0.4
    accept-encoding: gzip, deflate
    authorization: Basic <masked>
    user-agent: Splunk Website Input (+https://splunkbase.splunk.com/app/1818/)

Here is the HTTP request that comes from wget:

GET /all.xml HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: */*
Host: 192.168.0.41
Connection: Keep-Alive
Authorization: Basic <masked>

Does that make sense?

Thanks

0 Karma

jkat54
SplunkTrust
SplunkTrust

I dont think it has anything to do with the http header being authorization vs Authorization.

Do you have any error messages on the BBQ controller's logs?

Do you have any error messages in splunk logs? splunkd.log... index=_internal source=splunkd.log... etc?

Do you have any special characters in your password?

Can you post the wget command that works with the user/pass/servernames redacted?

Can you please verify that you get an authentication error when you use the web input app? IF you get another error message, please show it as well.

0 Karma

cmodyssey
Explorer

Hi,

Thanks for getting back to me.

I don't know if I can get the BBQ controller's logs.

Here is an extract of my splunk log, which is Website Input getting code 401 from my BBQ controller (Unauthorized):

12-28-2015 21:01:07.128 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunk/bin/splunkd instrument-resource-usage
12-28-2015 21:01:07.128 +0000 INFO  ExecProcessor -     interval: 0 ms
12-28-2015 21:01:07.134 +0000 INFO  ModularInputs - No stanzas found for scheme "web_input" in inputs.conf at script (re)start.
12-28-2015 21:01:07.135 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunk/bin/splunkd instrument-resource-usage
12-28-2015 21:01:07.135 +0000 INFO  ExecProcessor -     interval: 0 ms
12-28-2015 21:01:07.135 +0000 INFO  ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/website_input/bin/web_input.py
12-28-2015 21:01:07.135 +0000 INFO  ExecProcessor -     interval: run once
12-28-2015 21:01:11.576 +0000 ERROR HttpListener - Exception while processing request from 192.168.1.59 for /en-US/manager/search/data/inputs/web_input/CyberQ: Connection closed by peer
12-28-2015 21:01:11.577 +0000 ERROR HttpListener - Handler for /en-US/manager/search/data/inputs/web_input/CyberQ sent a 0 byte response after earlier claiming a Content-Length of 12336!
12-28-2015 21:01:15.200 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunk/bin/splunkd instrument-resource-usage
12-28-2015 21:01:15.201 +0000 INFO  ExecProcessor -     interval: 0 ms
12-28-2015 21:01:15.208 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunk/bin/splunkd instrument-resource-usage
12-28-2015 21:01:15.208 +0000 INFO  ExecProcessor -     interval: 0 ms
12-28-2015 21:01:15.208 +0000 INFO  ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/website_input/bin/web_input.py
12-28-2015 21:01:15.208 +0000 INFO  ExecProcessor -     interval: run once

My password currently only has letters in it.

Here is my working wget test:

[richard@flyer ~]$ wget --user=**** --password=******** http://192.168.0.4/all.xml
--2015-12-28 21:04:17--  http://192.168.0.4/all.xml
Connecting to 192.168.0.4:80... connected.
HTTP request sent, awaiting response... 401 Authorization Required
Connecting to 192.168.0.4:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/xml]
Saving to: “all.xml”

    [  <=>                                                                                                                                                                                               ] 1,235       4.11K/s   in 0.3s

2015-12-28 21:04:18 (4.11 KB/s) - “all.xml” saved [1235]

[richard@flyer ~]$ more all.xml
<nutcallstatus>
   <!--this is similar to status.xml, but with more values-->
   <!--all temperatures are displayed in tenths F, regardless of setting of unit-->
   <!--all temperatures sent by browser to unit should be in F.  you can send tenths F with a decimal place, ex: 123.5-->
   <COOK>
      <COOK_NAME>Cook</COOK_NAME>
      <COOK_TEMP>738</COOK_TEMP>
      <COOK_SET>3560</COOK_SET>
      <COOK_STATUS>0</COOK_STATUS>
   </COOK>
   <FOOD1>
      <FOOD1_NAME>Food1</FOOD1_NAME>
      <FOOD1_TEMP>OPEN</FOOD1_TEMP>
      <FOOD1_SET>1800</FOOD1_SET>
      <FOOD1_STATUS>4</FOOD1_STATUS>
   </FOOD1>
   <FOOD2>
      <FOOD2_NAME>Food2</FOOD2_NAME>
      <FOOD2_TEMP>OPEN</FOOD2_TEMP>
      <FOOD2_SET>1800</FOOD2_SET>
      <FOOD2_STATUS>4</FOOD2_STATUS>
   </FOOD2>
   <FOOD3>
      <FOOD3_NAME>Food3</FOOD3_NAME>
      <FOOD3_TEMP>OPEN</FOOD3_TEMP>
      <FOOD3_SET>1800</FOOD3_SET>
      <FOOD3_STATUS>4</FOOD3_STATUS>
   </FOOD3>
   <OUTPUT_PERCENT>100</OUTPUT_PERCENT>
   <TIMER_CURR>00:00:00</TIMER_CURR>
   <TIMER_STATUS>0</TIMER_STATUS>
   <DEG_UNITS>1</DEG_UNITS>
   <COOK_CYCTIME>6</COOK_CYCTIME>
   <COOK_PROPBAND>300</COOK_PROPBAND>
   <COOK_RAMP>0</COOK_RAMP>
</nutcallstatus>
[richard@flyer ~]$

My Splunk event log is full of the below, which is

response_size="407" raw_match_count="0" response_code="401" encoding="ascii" request_time="316.673994064"

What makes me think the problem is authorization vs Authorization are the following telnet tests:

Using request sent by wget and submitting via telnet:

[richard@flyer ~]$ telnet 192.168.0.4 80
Trying 192.168.0.4...
Connected to 192.168.0.4.
Escape character is '^]'.
GET /all.xml HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: */*
Host: 192.168.0.4
Connection: Keep-Alive
Authorization: Basic ********

HTTP/1.1 200 OK
Content-Type: text/xml
Cache-Control: no-cache
Connection: close

<nutcallstatus>
   <!--this is similar to status.xml, but with more values-->
   <!--all temperatures are displayed in tenths F, regardless of setting of unit-->
   <!--all temperatures sent by browser to unit should be in F.  you can send tenths F with a decimal place, ex: 123.5-->
   <COOK>
      <COOK_NAME>Cook</COOK_NAME>
      <COOK_TEMP>733</COOK_TEMP>
      <COOK_SET>3560</COOK_SET>
      <COOK_STATUS>0</COOK_STATUS>
   </COOK>
   <FOOD1>
      <FOOD1_NAME>Food1</FOOD1_NAME>
      <FOOD1_TEMP>OPEN</FOOD1_TEMP>
      <FOOD1_SET>1800</FOOD1_SET>
      <FOOD1_STATUS>4</FOOD1_STATUS>
   </FOOD1>
   <FOOD2>
      <FOOD2_NAME>Food2</FOOD2_NAME>
      <FOOD2_TEMP>OPEN</FOOD2_TEMP>
      <FOOD2_SET>1800</FOOD2_SET>
      <FOOD2_STATUS>4</FOOD2_STATUS>
   </FOOD2>
   <FOOD3>
      <FOOD3_NAME>Food3</FOOD3_NAME>
      <FOOD3_TEMP>OPEN</FOOD3_TEMP>
      <FOOD3_SET>1800</FOOD3_SET>
      <FOOD3_STATUS>4</FOOD3_STATUS>
   </FOOD3>
   <OUTPUT_PERCENT>100</OUTPUT_PERCENT>
   <TIMER_CURR>00:00:00</TIMER_CURR>
   <TIMER_STATUS>0</TIMER_STATUS>
   <DEG_UNITS>1</DEG_UNITS>
   <COOK_CYCTIME>6</COOK_CYCTIME>
   <COOK_PROPBAND>300</COOK_PROPBAND>
   <COOK_RAMP>0</COOK_RAMP>
</nutcallstatus>Connection closed by foreign host.
[richard@flyer ~]$

Using request sent by Website Input and submitting via telnet:

[richard@flyer ~]$ telnet 192.168.0.4 80
Trying 192.168.0.4...
Connected to 192.168.0.4.
Escape character is '^]'.
GET /all.xml HTTP/1.1
Host: 192.168.0.4
accept-encoding: gzip, deflate
authorization: Basic ********
user-agent: Splunk Website Input (+https://splunkbase.splunk.com/app/1818/)

HTTP/1.1 401 Authorization Required
WWW-Authenticate: Basic realm="Authorization Required"
Content-Type: text/html
Cache-Control: no-cache
Connection: close

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>401 Authorization Required</title>
</head><body>
<h1>Authorization Required</h1>
<p>This server could not verify that you are authorized to access the document requested.  Either you supplied the wrong  credentials (e.g., bad password), or your browser doesn't understand how to supply the credentials required.</p>
</body></html>Connection closed by foreign host.
[richard@flyer ~]$

Using request sent by Website Input, changing authorization to Authorization and submitting via telnet:

[richard@flyer ~]$ telnet 192.168.0.4 80
Trying 192.168.0.4...
Connected to 192.168.0.4.
Escape character is '^]'.
GET /all.xml HTTP/1.1
Host: 192.168.0.4
accept-encoding: gzip, deflate
Authorization: Basic ********
user-agent: Splunk Website Input (+https://splunkbase.splunk.com/app/1818/)

HTTP/1.1 200 OK
Content-Type: text/xml
Cache-Control: no-cache
Connection: close

<nutcallstatus>
   <!--this is similar to status.xml, but with more values-->
   <!--all temperatures are displayed in tenths F, regardless of setting of unit-->
   <!--all temperatures sent by browser to unit should be in F.  you can send tenths F with a decimal place, ex: 123.5-->
   <COOK>
      <COOK_NAME>Cook</COOK_NAME>
      <COOK_TEMP>733</COOK_TEMP>
      <COOK_SET>3560</COOK_SET>
      <COOK_STATUS>0</COOK_STATUS>
   </COOK>
   <FOOD1>
      <FOOD1_NAME>Food1</FOOD1_NAME>
      <FOOD1_TEMP>OPEN</FOOD1_TEMP>
      <FOOD1_SET>1800</FOOD1_SET>
      <FOOD1_STATUS>4</FOOD1_STATUS>
   </FOOD1>
   <FOOD2>
      <FOOD2_NAME>Food2</FOOD2_NAME>
      <FOOD2_TEMP>OPEN</FOOD2_TEMP>
      <FOOD2_SET>1800</FOOD2_SET>
      <FOOD2_STATUS>4</FOOD2_STATUS>
   </FOOD2>
   <FOOD3>
      <FOOD3_NAME>Food3</FOOD3_NAME>
      <FOOD3_TEMP>OPEN</FOOD3_TEMP>
      <FOOD3_SET>1800</FOOD3_SET>
      <FOOD3_STATUS>4</FOOD3_STATUS>
   </FOOD3>
   <OUTPUT_PERCENT>100</OUTPUT_PERCENT>
   <TIMER_CURR>00:00:00</TIMER_CURR>
   <TIMER_STATUS>0</TIMER_STATUS>
   <DEG_UNITS>1</DEG_UNITS>
   <COOK_CYCTIME>6</COOK_CYCTIME>
   <COOK_PROPBAND>300</COOK_PROPBAND>
   <COOK_RAMP>0</COOK_RAMP>

Does that explain my thinking?

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...