All Apps and Add-ons

Splunk Add-on for Microsoft Cloud Services: How to edit props.conf for Azure Storage Blob multi-event JSON ingestion?

pkeller
Contributor

Using the Splunk Add-On for Microsoft Cloud Services to pull Azure data ... I'm having some difficulty indexing Azure Storage Blobs ... Anyone have any suggestions for props.conf on the sourcetype? I'm currently doing this, but events are still being broken in random places.

[mscs:storage:blob]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\{
TRUNCATE = 0
KV_MODE = json

I'd welcome any better suggestions.

props.conf has been deployed to the heavy forwarder, all indexers, and the search head cluster (although I'm not sure that it's necessary there)

Splunk version HF - 6.4.3, Indexers - 6.4.3, Search 6.4.3

0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi pkeller,

just had the same troubles and made it work with this props.conf

[mscs:storage:blob]
KV_MODE = json
MAX_TIMESTAMP_LOOKAHEAD = 800
NO_BINARY_CHECK = true
category = Structured
description = A variant of the JSON source type, with support for Azure NGS blob format
disabled = false
pulldown_type = true
BREAK_ONLY_BEFORE_DATE = false
LINE_BREAKER = (,[\r\n]+\s+)\{
SEDCMD-mscs:storage:blob-StripHeader = \{\s+"records":\s+\[\s+

Deployed it to the indexers and the search heads.

Hope this helps ...

cheers, MuS

View solution in original post

0 Karma

aaraneta_splunk
Splunk Employee
Splunk Employee

@pkeller - Did the answer provided by MuS help provide a working solution to your question? If yes, please don't forget to resolve this post by clicking "Accept". If no, please leave a comment with more feedback. Thanks!

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi pkeller,

just had the same troubles and made it work with this props.conf

[mscs:storage:blob]
KV_MODE = json
MAX_TIMESTAMP_LOOKAHEAD = 800
NO_BINARY_CHECK = true
category = Structured
description = A variant of the JSON source type, with support for Azure NGS blob format
disabled = false
pulldown_type = true
BREAK_ONLY_BEFORE_DATE = false
LINE_BREAKER = (,[\r\n]+\s+)\{
SEDCMD-mscs:storage:blob-StripHeader = \{\s+"records":\s+\[\s+

Deployed it to the indexers and the search heads.

Hope this helps ...

cheers, MuS

0 Karma

hossyee
Engager

I think that it is necessary to delete the trailer as well.
[mscs:storage:blob]
SEDCMD-removetrailer = s/]\s+}//    << ADD

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...