What is the best practice performance-wise to get data in using C#?

[paulwrussell]

Hi,

My application is written in C#. I see there are a few ways for getting data in:

1. C# SDK (as per submit example)
2. Logging.net with HTTP Event Collector
3. Posting raw JSON to HTTP Event Collector
4. Somehow posting directly to a TCP input (I can't figure out how to do this)

I'm going to be indexing a lot of data and I'm trying to figure out the best method for doing this from my C# app in a threadsafe way. I've read through most of the documentation, but can't really tell what best practice is. It looks like HTTP EC may be the new recommended way. But is using logging .net really the most appropriate way of getting application data into Splunk? This seems more focused around getting application events into Splunk.

[gblock_splunk]

Hi @paulwrussell sorry for the delayed response.

HTTP Event Collector is our recommended way going forward for getting events from applications directly into Splunk. As far as whether to use the logging libraries or posting directly. The main advantage of our logging libs is we designed them robustly. They are async, they are fault tolerant / support retries. Not to mention they support the common .NET loggers that are out there i.e. anything that works with Trace Listeners or SLAB/ETW. Also you can use our shared lib to send JSON objects if you prefer to do that over string log messages.

As to the UF, we don't support UF on HEC. You can directly send to the indexer, OR you can use an HWF by enabling HEC on it. You can read more on our deployment options here.

[paulwrussell]

Thanks for the answer. Very helpful. Ive decided to go down route of using common logging and sending JSON directly. But if i do this, is there anyway of setting the timestamp? I'd like this option for passing in historical data.

[gblock_splunk]

Hi Paul

Great catch, we need to fix this. The timestamp property is currently readonly. If you want you could send us a PR to make it settable. Otherwise we have some other slight work we are going to do and we can handle this then.

[paulwrussell]

ok great thanks. i might try and add that when i get some time. one last question. when i use automatic batching, will the send method act in an asynchronous manner? or for this, would i have to do manual batching?

[gblock_splunk]

No problem

No you do not have to do manual batching. Send is synchronous because all it does it write to a queue where a background reader pulls it. It will by default send sequentially though async. This is to ensure that all the data is sent in order. You can configure it to not send sequentially by setting sendMode. If you use any sort of batching and want to force it send the batch before any of the batch limits (size, etc) have been hit, you can force by calling FlushAsync / FlushSync.

[jkat54]

I believe logging.net is a best practice because it "leaves" a log file somewhere which can be re-ingested if need be.

I guess the idea is to keep a copy of the data for best practice.

Also, the universal forwarder, and forwarder capabilities of splunk are coded "much better" than you or I could code. So I would say write to file, use the forwarder, get all the advantages of having a forwarder:

1. Load balancing
2. Fishbucket
3. Auditing/Analyitics
4. Throttling
5. etc.

[paulwrussell]

Thanks for the response.

If i couldn't use the universal forwarder, what is the recommended best approach for high throughput?