All Apps and Add-ons

Splunk Add-on for Microsoft Cloud Services: How to configure blob_mode for Azure Storage Blob Modular Input to not check for changes after indexing?

nbouchia
New Member

Is it possible to have another value for "blob_mode" than "append" or to index the blob one time and after do not check changes?
We write in a blob one time, and after that, we do not make changes. However, Splunk continues to check if there's a modification.
We have more than 100 000 blobs and now it's affecting performance having to check all these blobs.

0 Karma

lding_splunk
Splunk Employee
Splunk Employee

Hi nbouchia,
thanks for the proposal, actually the add-on DO NOT touch the file again once it's indexed and have no further changes.
I think it's related tot he large count of blob files, but would you please provide with more detail about the performance issue you encountered?

0 Karma

nbouchia
New Member

Hello

We have an application that's write in a different blob at each instantiation. We have around 200 000 blobs in the container.
When a blob is created after it doesn't change.
With the add-on we pull the container that contains all these blob. We configure the add-on to pull every 5mn.
I thinks our problem performance occured when the add-on test the last modified time for all blobs to see if there's new data to index.

0 Karma

lding_splunk
Splunk Employee
Splunk Employee

Hi nbouchia
thanks for the reply, may i know the the performance as it is in your env? say, how much delay or lag? OTOH, if you could provide some screenshot as well diag files (by reference Splunk diag), that will be very helpful. Thanks!

0 Karma

nbouchia
New Member

Hello
We have one server (indexer+search head in the same server).
Initialy, the delay was put to 60sec but now we put it to 10mn.
And now, we archive the blobs, so we don't have the same problem but with time it will happen again.
We showed the problem with a lot of I/O in our system and some crashed.

0 Karma

lding_splunk
Splunk Employee
Splunk Employee

Hi nbouchia
thanks for the reply, it will be great if you could contact splunk support by attaching those diag files. Thanks!

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...