All Apps and Add-ons

How to configure csv sourcetype "mscs:storage:blob:csv" for Splunk add-on for MS Cloud Services?

mlevsh
Builder

We are trying to Configure Azure Storage Blob Modular Inputs for Splunk Add-on for Microsoft Cloud Services to get reports, that come in csv format. We have created props.conf TA folder in /opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/local folder with the following sourcetype stanza and still field extraction is not working. Any advices?

[mscs:storage:blob:csv]
BREAK_ONLY_BEFORE_DATE =
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false

Thank you!

Labels (1)
0 Karma

VatsalJagani
Champion

@mlevsh - As mentioned by @mbjerkeland_spl, INDEXED_EXTRACTION will not work. Use search-time CSV extraction.

# props.conf
[data]
REPORT-headers = data_headers


# transforms.conf
[data_headers]
CLEAN_KEYS = 0
DELIMS = ","
FIELDS = <comma-separated-field-list>

 

I hope this helps!!!

0 Karma

mbjerkeland_spl
Splunk Employee
Splunk Employee

INDEXED_EXTRACTIONS = CSV is not supported by modular inputs according to https://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileswithstructureddata

Input types that the indexed field extraction feature supports

This feature works with the following input types:

  • File-based inputs only (such as monitoring files, directories, or archives.)
  • Inputs that use the oneshot input type (or through the "Upload" feature in Splunk Web.)

It does not work with modular inputs, network inputs, or any other type of input.

 

You should instead use delimited field extractions to achieve the same result. See: 

  1. https://docs.splunk.com/Documentation/SplunkCloud/8.2.2203/Knowledge/FXSelectMethodstep
  2. https://www.splunk.com/en_us/blog/tips-and-tricks/quick-n-dirty-delimited-data-sourcetypes-and-you.h...

 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

It would help to see sample input (sanitized as necessary).

Where did you put the props.conf file?  Did you restart Splunk after modifying the file?  Does the data come in to the indexers directly or via a heavy forwarder?

---
If this reply helps you, an upvote would be appreciated.

mlevsh
Builder

Hi @richgalloway 

Sorry for the delay. Hopefully you will see my reply

1) Where did you put the props.conf file?
in app local directory: /opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/local

2)  Did you restart Splunk after modifying the file? 
Yes, splunk was restarted

3) Does the data come in to the indexers directly or via a heavy forwarder? 
Best to my knowledge, they are going to heavy forwarder first.

Input is csv file with a lot of columns (from A to BA) , first line is a header. When I onboard with the same props.conf via Data input right on Heavy Forwarder - it extracts fields perfectly. 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Here are a few things to verify.

  1. The props.conf file is installed on the heavy forwarder.
  2. In case data goes to the indexer directly rather than through the HF, install the props.conf on the indexer as well.
  3. The sourcetype is set correctly in inputs.conf.
---
If this reply helps you, an upvote would be appreciated.

mlevsh
Builder

@richgalloway   Thank you for your reply! I will verify if data are going directly to the heavy forwarder or indexers . We have Heavy Forwarder - on - prem , but search head and indexers are on Splunk Cloud.  So we installed  and configured Splunk add-on for MS Cloud Services on our on-prem Heavy Forwarder and assumed that Azure storage blob data are being pulled from our Heavy Forwarder
 

I also see the following error messages in internal logs:


Unable to find segmenter for conf=source::source_storage_blob_name.csv|host::our_heavy_forwarder|mscs:storage:blob:csv|remoteport::17575. Will attempt to use the default configuration

0 Karma