We are trying to Configure Azure Storage Blob Modular Inputs for Splunk Add-on for Microsoft Cloud Services to get reports, that come in csv format. We have created props.conf TA folder in /opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/local folder with the following sourcetype stanza and still field extraction is not working. Any advices?
[mscs:storage:blob:csv]
BREAK_ONLY_BEFORE_DATE =
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
Thank you!
@mlevsh - As mentioned by @mbjerkeland_spl, INDEXED_EXTRACTION will not work. Use search-time CSV extraction.
# props.conf
[data]
REPORT-headers = data_headers
# transforms.conf
[data_headers]
CLEAN_KEYS = 0
DELIMS = ","
FIELDS = <comma-separated-field-list>
I hope this helps!!!
INDEXED_EXTRACTIONS = CSV is not supported by modular inputs according to https://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileswithstructureddata
This feature works with the following input types:
It does not work with modular inputs, network inputs, or any other type of input.
You should instead use delimited field extractions to achieve the same result. See:
It would help to see sample input (sanitized as necessary).
Where did you put the props.conf file? Did you restart Splunk after modifying the file? Does the data come in to the indexers directly or via a heavy forwarder?
Sorry for the delay. Hopefully you will see my reply
1) Where did you put the props.conf file?
in app local directory: /opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/local
2) Did you restart Splunk after modifying the file?
Yes, splunk was restarted
3) Does the data come in to the indexers directly or via a heavy forwarder?
Best to my knowledge, they are going to heavy forwarder first.
Input is csv file with a lot of columns (from A to BA) , first line is a header. When I onboard with the same props.conf via Data input right on Heavy Forwarder - it extracts fields perfectly.
Here are a few things to verify.
@richgalloway Thank you for your reply! I will verify if data are going directly to the heavy forwarder or indexers . We have Heavy Forwarder - on - prem , but search head and indexers are on Splunk Cloud. So we installed and configured Splunk add-on for MS Cloud Services on our on-prem Heavy Forwarder and assumed that Azure storage blob data are being pulled from our Heavy Forwarder
I also see the following error messages in internal logs:
Unable to find segmenter for conf=source::source_storage_blob_name.csv|host::our_heavy_forwarder|mscs:storage:blob:csv|remoteport::17575. Will attempt to use the default configuration