All Apps and Add-ons

Index gz archive files from Storage Account via Splunk Add-on for Microsoft Cloud Services not working

Path Finder

Is it possible to index gz archives files from a Azure Storage account into Splunk using the Splunk Add-on for Microsoft Cloud Services

Import of not archived files via the Splunk Add-on for Microsoft Cloud Services is working perfect. I already tested using the following parameters for source and/or sourcetype but Splunk is still import only "weird symbols"/the raw gz file.

props.conf (tested within source + sourcetype stanza)
invalidcause = archive
unarchive
cmd = gzip -cd -

I also tested to NOBINARYCHECK = false to explicitly ignore the gz files (which is not really my goal) but also that doesn't work at all. Downloading the gz file manually and import into Splunk manually is working fine. So I assume that issue is related to the way the Splunk Add-on for Microsoft Cloud Services is consuming and forwarding the data. Hopefully someone else was running into the same issue/able to fix that issue.

Thanks for your help in advance,
Thorsten

1 Solution

Splunk Employee
Splunk Employee

It sounds like you are using the blob input in the Splunk Add-on for Microsoft Cloud Services. The short answer to your question is the add-on will not decompress the archive to read the contents.

Here is the technical answer:
The add-on doesn't actually download the blob data and then read the content. Instead, the blob input uses the Azure Python SDK to get a blob's content into text. Specifically, it uses the getblobto_text method in the blob service -> https://github.com/Azure/azure-storage-python/blob/master/azure-storage-blob/azure/storage/blob/base... This method streams blob bytes as text. So, when encountering a binary file (like a .gz archive file), you get the weird characters.

Depending on your use case, it is possible to download .gz files from Azure blobs that can then be indexed using traditional file methods. A tool like AzCopy can be handy for this -> https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy

View solution in original post

Splunk Employee
Splunk Employee

It sounds like you are using the blob input in the Splunk Add-on for Microsoft Cloud Services. The short answer to your question is the add-on will not decompress the archive to read the contents.

Here is the technical answer:
The add-on doesn't actually download the blob data and then read the content. Instead, the blob input uses the Azure Python SDK to get a blob's content into text. Specifically, it uses the getblobto_text method in the blob service -> https://github.com/Azure/azure-storage-python/blob/master/azure-storage-blob/azure/storage/blob/base... This method streams blob bytes as text. So, when encountering a binary file (like a .gz archive file), you get the weird characters.

Depending on your use case, it is possible to download .gz files from Azure blobs that can then be indexed using traditional file methods. A tool like AzCopy can be handy for this -> https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy

View solution in original post

Path Finder

Thanks Jason for clarification. Your answer perfectly explain why I was running into a another issue when trying to download a CSV via this app and using the Splunk capabilities of indexing CSV files didn't work. I was able to find a workaround for this ignoring the first line and using a key/value extraction at search time so not a big deal in that case.

Just to make sure I got your right, you recommend, at least for now, to download the files manually and than index the data using a filter/folder monitor input?

0 Karma