Getting Data In

How to resolve when a C# HttpWebRequest to REST API is not working?

RhinoTX
Explorer

Hey there, trying to use C# WebRequest class to send a simple search (via POST to get the sid then GET to get the results) and I can't get the first step to work. I either seem to get 401 Not Authorized (but it works via curl cmdline), or 400 Bad Request, depending on how I pass the authentication. I've tried every combination I can think of. Oh, and using the SDK is not an option.

Here's the code that posts the search, using my custom class WebAPI (a simple wrapper) which I will provide code for later:

                WebAPI api = new WebAPI();
                api.Trust(); // we trust our local splunk server (this disables SSL certificate checking!!!)

                string username = "<removed>";
                string password = "<removed>";

                string scheme = "https";
                string address = "169.00.000.168";
                uint port = 2711;
                string path = "/services/search/jobs";
                string post_content = "search=\"search index=mmogbrain sourcetype=dreadnought_profile host=DN-GL-MM-000 earliest=-1m@m latest=now|table _time,connections\"";

                string url = api.BuildUrl(scheme, address, port, path, null);

                string result = api.Post(url, content, username, password);

Here's the code for WebAPI helper class:

using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Text;
using System.Web.Script.Serialization;

namespace SixFoot.Core
{
    public class WebAPI
    {
        public void Trust()
        {
            ServicePointManager.ServerCertificateValidationCallback = ((sender, certificate, chain, sslPolicyErrors) => true);
        }

        public void Post(string url, string content)
        {
            Post(url, content, null, null);
        }

        public string Post(string url, string content, string username, string password) 
        {
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
            request.Method = "POST";

            System.Text.UTF8Encoding encoding = new System.Text.UTF8Encoding();
            Byte[] byteArray = encoding.GetBytes(content);

            request.ContentLength = byteArray.Length;


            if (username != null && password != null)
            {
                string authInfo = username + ":" + password;
                authInfo = Convert.ToBase64String(Encoding.GetEncoding("ISO-8859-1").GetBytes(authInfo));
                request.Headers["Authorization"] = "Basic " + authInfo;
            }

            using (Stream dataStream = request.GetRequestStream())
            {
                dataStream.Write(byteArray, 0, byteArray.Length);
            }

            try
            {
                WebResponse response = request.GetResponse();
                using (Stream responseStream = response.GetResponseStream())
                {
                    StreamReader reader = new StreamReader(responseStream, Encoding.UTF8);
                    return reader.ReadToEnd();
                }

            }
            catch (WebException ex)
            {
                string message = ex.Message;

                WebResponse errorResponse = ex.Response;
                if (errorResponse != null)
                {
                    using (Stream responseStream = errorResponse.GetResponseStream())
                    {
                        StreamReader reader = new StreamReader(responseStream, Encoding.GetEncoding("utf-8"));
                        message = reader.ReadToEnd();                    
                    }
                }

                throw;
            }
        }

        public Dictionary<string, object> ParseJson(string json)
        {
            JavaScriptSerializer serializer = new JavaScriptSerializer();
            return (Dictionary<string, object>) serializer.DeserializeObject(json);
        }

        public string BuildUrl(string scheme, string address, uint port, string path, string query)
        {
            UriBuilder builder = new UriBuilder(scheme, address, (int) port);
            builder.Path = path;
            builder.Query = query;

            return builder.Uri.AbsoluteUri;
        }

        // snip removed similar Get methods...but I'm no getting that far. ;-(

    }
}

Any thoughts? The above code returns "401 Bad Request". Posting the exact same strings via curl works and returns a sid.

Thanks in advance for any help...

0 Karma
1 Solution

RhinoTX
Explorer

I figured this out. 😉

The bug is here:

string post_content = "search=\"search index=mmogbrain sourcetype=dreadnought_profile host=DN-GL-MM-000 earliest=-1m@m latest=now|table _time,connections\"";

The search query has double-quotes surrounding the search value. This trips up the Splunk parser for some reason. The search as provided by the splunk admin was based on him running it in CURL, and in CURL of course since it's a cmdline it must have the quotes since ther are spaces in the search string. Those quotes need to be removed when sending to the server. Once I did that, all started working.

View solution in original post

RhinoTX
Explorer

I figured this out. 😉

The bug is here:

string post_content = "search=\"search index=mmogbrain sourcetype=dreadnought_profile host=DN-GL-MM-000 earliest=-1m@m latest=now|table _time,connections\"";

The search query has double-quotes surrounding the search value. This trips up the Splunk parser for some reason. The search as provided by the splunk admin was based on him running it in CURL, and in CURL of course since it's a cmdline it must have the quotes since ther are spaces in the search string. Those quotes need to be removed when sending to the server. Once I did that, all started working.

RhinoTX
Explorer

Bumpity Bump....

0 Karma

RhinoTX
Explorer

BUMP!!!!

I am STILL stuck on this...

I have rewritten the code three times using various HTTP connection methods and STILL always get 400 Bad Request. Again, it WORKS FROM CURL!!! So I know what I'm sending is a valid search. I can't even get auth login to work!!! 400 Bad Request... There is NOTHING USEFUL coming from the Splunk logs other than the fact that a bad request was received. ;-( There is nothing, that I can see, that tells me HOW it was a malformed request (if it even is...)

Here's the latest version using simple WebClient which from other samples I've seen on the Internet SHOULD work...

var client = new WebClient();
var content = WebUtility.UrlEncode(string.Format("{{'username':{0}, 'password':{1}}}", username, password));
var url = "https://:/services/auth/login";
client.Headers["Content-Type"] = "application/json";

var result = client.UploadString(url, content); // THIS THROWS 400 Bad Request

I am at a loss. I have tried this using HttpWebRequest. I have tried this using WebClient. I even used an old http client impl I had written directly using sockets. All are receive 400 Bad Request from the server and I have ZERO CLUE WHY and your server provides ZERO WAY TO DIAGNOSE THIS!!!!!!

Again, it works from curl.exe, using the exact same ip, port, and path.

I think a reply is overdue here folks, it's been days...

Using the C# SDK is not an option...this should work!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...