All Apps and Add-ons

Splunk Add-on for Microsoft Cloud Services: How to configure blob_mode for Azure Storage Blob Modular Input to not check for changes after indexing?

nbouchia
New Member

Is it possible to have another value for "blob_mode" than "append" or to index the blob one time and after do not check changes?
We write in a blob one time, and after that, we do not make changes. However, Splunk continues to check if there's a modification.
We have more than 100 000 blobs and now it's affecting performance having to check all these blobs.

0 Karma

lding_splunk
Splunk Employee
Splunk Employee

Hi nbouchia,
thanks for the proposal, actually the add-on DO NOT touch the file again once it's indexed and have no further changes.
I think it's related tot he large count of blob files, but would you please provide with more detail about the performance issue you encountered?

0 Karma

nbouchia
New Member

Hello

We have an application that's write in a different blob at each instantiation. We have around 200 000 blobs in the container.
When a blob is created after it doesn't change.
With the add-on we pull the container that contains all these blob. We configure the add-on to pull every 5mn.
I thinks our problem performance occured when the add-on test the last modified time for all blobs to see if there's new data to index.

0 Karma

lding_splunk
Splunk Employee
Splunk Employee

Hi nbouchia
thanks for the reply, may i know the the performance as it is in your env? say, how much delay or lag? OTOH, if you could provide some screenshot as well diag files (by reference Splunk diag), that will be very helpful. Thanks!

0 Karma

nbouchia
New Member

Hello
We have one server (indexer+search head in the same server).
Initialy, the delay was put to 60sec but now we put it to 10mn.
And now, we archive the blobs, so we don't have the same problem but with time it will happen again.
We showed the problem with a lot of I/O in our system and some crashed.

0 Karma

lding_splunk
Splunk Employee
Splunk Employee

Hi nbouchia
thanks for the reply, it will be great if you could contact splunk support by attaching those diag files. Thanks!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...