I am working on seeing if there is a better way that I can consume the data I have from an Azure Blob storage.
I am using the Splunk Addon for Microsoft Cloud Services; which is allowing me to grab the file from the Blob storage.
Now the file itself is actually a CSV; but the app does not recognize CSV to split up the data so I had to create a props.conf and transforms.conf which does allow me to get the data in and into fields. But it turns out that the CSV fields has 1 field in it that is the rawMessage and it is actually a JSON of all the field data (and sometimes will contain a field which is not broken into a column of the CSV.
For now I am bringing all of the data in and separated into fields and then at search time doing a spath on the rawMessage field to get the other fields that only sometimes appear.
What I would like to do is instead of all of the data being brought in; I want to only bring in the rawMessage field (which is JSON) and have that indexed as it has all of the data/fields.
What is the best way to write the inputs/props/transforms to only read that field in and then parse it as a JSON so it will do the autobreakdown for fields?
Any help with this would be greatly appreciated.
