Use of Splunk C# API under heavy traffic

[wdthem]

Hi there,

Please forgive the questions from a Splunk n00b, but I have not been able to find what I need thus far.

I am writing an A/B testing framework for our site. Originally we thought we'd use StatsD to emit logging data from the framework for our analyses. But, we already have and love Splunk and it makes the most sense to use it instead, and make use of its statistical analysis capabilities.

So, with the C# API, my questions are:

1. Should there only be one instance of the Service for the lifetime of the .NET App? Is the Service object threadsafe or do I need to handle that manually?
2. What sorts of traffic can be thrown at the Service? I will want to be sending the equivalent of StatsD's "Counter" data (just log a +1 next to an event). On busy pages we can implement sample rating (e.g. 10% of requests), but there is still the potential for a high volume of calls going to Splunk. What are the implications here?
3. If Splunk were to go down while the API is sending requests, will the Service handle this or will it cause the .NET app to hang? Is the Service using a fire-and-forget method or not?

Also - should I be using the Service object or the WebClient object?

Apologies for the list of questions - help is most appreciated!

Thanks.

-Will

[wdthem]

Do either of you have an example of connecting to the Forwarder over UDP? In the SDK I am only seeing a "Receiver" object.

Or am I missing the point and do I just make a new .NET UdpClient and send data that way - assuming the forwarder is configured properly to accept the data?

[ywu_splunk · Accepted solution]

Hi Will

Below are some answers:

1. Service class is not thread safe. To make multiple requests/calls to Splunk in parallel, multiple Service objects should be created and used. There's no requirement of having only one instance of Service object for the lifetime of the .NET App. However, reusing the Service object can benefit performance and scale. You can authenticate once and service object will cache and reuse the auth token.

2. It seems that you'd like to log data into Splunk using the SDK. That is indeed supported. Take a look at the "submit" example included in the SDK. It submits events into Splunk using HTTP. For better performance and throughput, you may also using raw socket directly. That is also supported by the SDK. You attach a Stream object to an Index object or a Receiver object and then write data into stream directly. Refer to UnitTests.IndexTest.IndexEvents.

3. The SDK will not handle this automatically. It will not hang however. It may take a while for the connection to report an error, which will result in an exception. The application may need to resubmit some events in that case. It is ok for the same event to be logged more than once. Splunk search has a dedup command which can remove duplicates easily.

What WebClient object were you referring to? We recommend you to use the SDK. It is a lot easier to use than to programming against Splunk REST API using .NET directly.

Thanks,
-Andy

[wdthem]

Thank you both, this is very helpful info!

[ineeman]

A couple of quick additions to Andy's answer:

If you want a more reliable way of getting data into Splunk, I would suggest using a Universal Forwarder. It will handle things like backpressure from Splunk and can also do more load balancing for you (for example, if you have multiple indexers in your data tier).

You can still use the SDK in that case: just connect to the forwarder over TCP/UDP. You can also just use the SDK and write the data to a file, and have the forwarder pick it up.

[paulwrussell]

@Andy: you said " For better performance and throughput, you may also using raw socket directly. That is also supported by the SDK. You attach a Stream object to an Index object or a Receiver object and then write data into stream directly. Refer to UnitTests.IndexTest.IndexEvents." Do you have example of posting using raw socket directly using C#? I cant find the mentioned test class.

Im going to be indexing alot of data. And I'm trying to figure out the best method for doing this from my C# app in a threadsafe way.