How to index .zip from Azure blob Storage via Splunk Add-on for Microsoft Cloud Services

[yanjingz]

Dear team,

Is there any recommended way to index .zip from Azure blob Storage via Splunk Add-on for Microsoft Cloud Services?

If it is impossible directly, is there any preferred workaround to unzip it someway? 

Big thanks!!!

[mattymo · Accepted solution]

At least as of the time of this comment, the docs say "No"

The Azure Storage Blob modular input for Splunk Add-on for Microsoft Cloud Services does not support the ingestion of gzip files. Only plaintext files are supported.

[yanjingz]

By the way, is there any workaround to unzip it? Will be really appreciated! 

[mattymo]

See my other comment. You will need another input method. Suggest you google Azure functions "unzip" and see if they can just use Azure to do that. Otherwise you would need custom code or scripted input to pull in the zip and pass to something like the `unarchive_cmd`

unarchive_cmd = <string>
* Only called if invalid_cause is set to "archive".
* This field is only valid on [source::<source>] stanzas.
* <string> specifies the shell command to run to extract an archived source.
* Must be a shell command that takes input on stdin and produces output on
  stdout.
* Use _auto for Splunk software's automatic handling of archive files (tar,
  tar.gz, tgz, tbz, tbz2, zip)
* This setting applies at input time, when data is first read by Splunk
  software, such as on a forwarder that has configured inputs acquiring the
  data.
* Default: empty string

Azure functions is likely the more scalable/flexible option, but if this is not a large amount of data, you might be able to hack together HF(s) to do this. 

Please, accept my original comment as solution to your post and review the options I mentioned in my comment. Also be sure to check out internal azure sme channels to learn more or holler at Pro Serv. 

[yanjingz]

Thanks, understood!

I will have to somehow unzip it first...

[mattymo]

Yeah, maybe investigate Azure Functions, pick up unzip, post to new blob, or send to HEC. Or HF and investigate a custom input to feed the `unarchive_cmd`

Make sure to accept the answer to original post if it was helpful. Thanks!