All Apps and Add-ons

Splunk Add-on for Microsoft Cloud Services: How to configure blob_mode for Azure Storage Blob Modular Input to not check for changes after indexing?

nbouchia
New Member

Is it possible to have another value for "blob_mode" than "append" or to index the blob one time and after do not check changes?
We write in a blob one time, and after that, we do not make changes. However, Splunk continues to check if there's a modification.
We have more than 100 000 blobs and now it's affecting performance having to check all these blobs.

0 Karma

lding_splunk
Splunk Employee
Splunk Employee

Hi nbouchia,
thanks for the proposal, actually the add-on DO NOT touch the file again once it's indexed and have no further changes.
I think it's related tot he large count of blob files, but would you please provide with more detail about the performance issue you encountered?

0 Karma

nbouchia
New Member

Hello

We have an application that's write in a different blob at each instantiation. We have around 200 000 blobs in the container.
When a blob is created after it doesn't change.
With the add-on we pull the container that contains all these blob. We configure the add-on to pull every 5mn.
I thinks our problem performance occured when the add-on test the last modified time for all blobs to see if there's new data to index.

0 Karma

lding_splunk
Splunk Employee
Splunk Employee

Hi nbouchia
thanks for the reply, may i know the the performance as it is in your env? say, how much delay or lag? OTOH, if you could provide some screenshot as well diag files (by reference Splunk diag), that will be very helpful. Thanks!

0 Karma

nbouchia
New Member

Hello
We have one server (indexer+search head in the same server).
Initialy, the delay was put to 60sec but now we put it to 10mn.
And now, we archive the blobs, so we don't have the same problem but with time it will happen again.
We showed the problem with a lot of I/O in our system and some crashed.

0 Karma

lding_splunk
Splunk Employee
Splunk Employee

Hi nbouchia
thanks for the reply, it will be great if you could contact splunk support by attaching those diag files. Thanks!

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...