Reading data from Azure Storage Table

[jeffbat]

I am trying to read data from an Azure Storage Table and currently am using the Splunk Add-on for Microsoft Cloud Services.

I am able to get the data read into Splunk for the whole table but am having trouble trying to get the host changed from the server where the data input runs and instead using part of one of the fields in the data being read in. (I want this done at index time)

The data in the Azure table is being written with NLog.

When the data is read in, Splunk recognizes multiple fields from the data in the columns. The field Message is json and inside there is a field of machine. That is what I am trying to get the host to be.

This is what I have in the .conf files:

inputs.conf

[mscs_storage_table://Test Table Read 10]
account = Testing POS Logs
collection_interval = 300
index = azure
sourcetype = mscs:storage:table:test10
start_time = 2018-04-17T16:00:09-07:00
table_list = POSNlog

props.conf

[mscs:storage:table:test10]
TRANSFORMS-host_rename=rename_host_by_field_host

transforms.conf

[rename_host_by_field_host]
SOURCE_KEY=field:Message
REGEX=Message="machine\":\"(?.+?(?=\"))"
FORMAT = host::$1
DEST_KEY=MetaData:Host

One of the entries being read in as indexed right now looks like this:

{"odata.etag": "W/\"datetime'2018-04-18T18%3A04%3A37.9493312Z'\"", "PartitionKey": "20180418.NLogAzureTest.Test2", "Timestamp": "2018-04-18T18:04:37.9493312Z", "Message": "{\"time\":\"2018-04-18 11:04:33.8902\",\"utc-time\":\"2018-04-18 18:04:33.8902\",\"level\":\"Error\",\"message\":\"Oh noes!\",\"exception\":\"System.ArgumentException: Too much boom!\r\n at NLogAzureTest.Test2.Log() in C:\\Users\\fischja\\Documents\\Visual Studio 2017\\Projects\\NLogAzureTest\\Program.cs:line 78\",\"exceptionData\":\"boomPercent: 100.10\",\"logger\":\"NLogAzureTest.Test2\",\"machine\":\"LT-B02107\",\"processId\":\"7924\",\"processName\":\"NLogAzureTest\",\"identity\":\"notauth::\",\"windowsIdentity\":\"TBECU\\fischja\"}", "RowKey": "0636596714738902451.0c653fa7-c116-4ba5-a3f5-327f7aebeb6f"}

Any ideas why I am not getting the host converted correctly?

Also a slightly different question about reading from the Azure Storage Tables. On the table we are reading from, we actually on care about the data in the Message field. Is there a way either with this app or something different to just pull in that field and part the data as straight json as that field is that way?

Thanks.

[jconger]

Try this REGEX in your transforms.conf

\"Message\"\:.+?machine\\"\:\\\"([^\\]+)

To answer your second question, you could use a couple of SEDCMDs to find and replace the stuff you don't want.

[p_gurav]

Can you try :

[rename_host_by_field_host]
REGEX=Message="machine\\\":\\\"([^\\]+)"
FORMAT = host::$1
DEST_KEY=MetaData:Host

[jeffbat]

Unfortunately, that still didn't work.