Getting Data In

Reading data from Azure Storage Table

Path Finder

I am trying to read data from an Azure Storage Table and currently am using the Splunk Add-on for Microsoft Cloud Services.

I am able to get the data read into Splunk for the whole table but am having trouble trying to get the host changed from the server where the data input runs and instead using part of one of the fields in the data being read in. (I want this done at index time)

The data in the Azure table is being written with NLog.

When the data is read in, Splunk recognizes multiple fields from the data in the columns. The field Message is json and inside there is a field of machine. That is what I am trying to get the host to be.

This is what I have in the .conf files:


[mscs_storage_table://Test Table Read 10]
account = Testing POS Logs
collection_interval = 300
index = azure
sourcetype = mscs:storage:table:test10
start_time = 2018-04-17T16:00:09-07:00
table_list = POSNlog




FORMAT = host::$1

One of the entries being read in as indexed right now looks like this:

{"odata.etag": "W/\"datetime'2018-04-18T18%3A04%3A37.9493312Z'\"", "PartitionKey": "20180418.NLogAzureTest.Test2", "Timestamp": "2018-04-18T18:04:37.9493312Z", "Message": "{\"time\":\"2018-04-18 11:04:33.8902\",\"utc-time\":\"2018-04-18 18:04:33.8902\",\"level\":\"Error\",\"message\":\"Oh noes!\",\"exception\":\"System.ArgumentException: Too much boom!\r\n at NLogAzureTest.Test2.Log() in C:\\Users\\fischja\\Documents\\Visual Studio 2017\\Projects\\NLogAzureTest\\Program.cs:line 78\",\"exceptionData\":\"boomPercent: 100.10\",\"logger\":\"NLogAzureTest.Test2\",\"machine\":\"LT-B02107\",\"processId\":\"7924\",\"processName\":\"NLogAzureTest\",\"identity\":\"notauth::\",\"windowsIdentity\":\"TBECU\\fischja\"}", "RowKey": "0636596714738902451.0c653fa7-c116-4ba5-a3f5-327f7aebeb6f"}

Any ideas why I am not getting the host converted correctly?

Also a slightly different question about reading from the Azure Storage Tables. On the table we are reading from, we actually on care about the data in the Message field. Is there a way either with this app or something different to just pull in that field and part the data as straight json as that field is that way?


0 Karma

Splunk Employee
Splunk Employee

Try this REGEX in your transforms.conf


To answer your second question, you could use a couple of SEDCMDs to find and replace the stuff you don't want.

0 Karma


Can you try :

FORMAT = host::$1
0 Karma

Path Finder

Unfortunately, that still didn't work.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!