Splunk Add-on for Microsoft Azure: When configuring storage account inputs, do I write out the full URL or the hostname?

[brent_weaver]

I am having marginal success. We are writing out Linux VMs syslogs to table storage, which I can see with Azure Storage explorer but does not show up in Splunk after having added this table to storage account inputs. For storage account, do I put in the full URL or just the hostname?

Does anyone have experience with this?

[brent_weaver]

I used this procedure to resolve this issue. It seems that somehow the input got checkpointed into 2021?!?!

Remove checkpoints
You can remove checkpoints by running the Splunk clean utility.

Caution: Be careful when removing checkpoints. Running the clean command removes your indexed data. For example, clean all removes ALL your indexed data.

For example, to remove checkpoints for a specific scheme:

splunk clean inputdata [<scheme>]

So to resolve this I ran:

splunk clean inputdata AzureTableStorage

This removed the check point file and splunk was in turn able to reset the checkpoint and carry on.

Thank you!

[jconger · Accepted solution]

The logic for the start date on the input works like this:

• Set a start date to 2 days ago. You'll see this in the debug message "AzureStorageTable:264 - dateTimeStart = '2016-09-14T14:11:56.740547'"
• Look for start date in inputs.conf. If we find one, use it. Otherwise use the 2 days ago one. It looks like you don't have a start date specified, so you get the debug message "No dateTimeStart in inputs.conf..."
• Look for a marker from the check point directory (which holds the last date/time we saw when looping through table rows). If there is a marker, then use the date/time from the marker. Your data has a marker as evidenced by this message "DEBUG AzureStorageTable:290 - Marker found for table LinuxsyslogVer2v0: 2028-06-17T20:31:07.379676+00:00"

When looping through the table rows, we look at the field specified for the timestamp to see if it is greater than any other row (basically keeping a variable with the largest date/time seen in the data). After looping through, we write this value to the checkpoint to use next time. So, it looks like your data has an invalid time stamp in Azure (2028-06-17T20:31:07.379676+00:00).

You can clear the checkpoint, but it will be beneficial to use the Azure Storage Explorer to see where this errant data is logged.

[brent_weaver]

I think this has been resolved by clearing the checkpoint file. I am doing more testing but it does seem to be getting logs. I will conclude this post with what I did to fix it if it does in fact work.

THANK YOU!

[brent_weaver]

THANK YOU so much for the great information!

I have already verified that the datetime in azure is correct.
No date in inputs.conf

Where is the checkpoint file? I cannot seem to find it even after having looked in the python script. Can this value be reset?

[brent_weaver]

The issue seems to be that the date ranges are being picked as 2021-xx-xx?!?

If I look at the log file I see this:

09-16-2016 14:11:56.741 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,741 DEBUG AzureStorageTable:299 - Filter string: ***TIMESTAMP gt datetime'2028-06-17***T20:31:07.379676+00:00' and TIMESTAMP lt datetime'2028-06-20T08:31:07.379676+00:00'

Why is this happening? ENabled debug in the python script:

/opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py

Any advice is MUCH appreciated!

[brent_weaver]

More information:

Here is from the log file:

09-16-2016 14:11:56.653 +0000 DEBUG ExecProcessor - ExecProcessorSharedState::addToRunQueue() path='python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py' restartTimerIfNeeded=1
09-16-2016 14:11:56.653 +0000 DEBUG ExecProcessor - adding "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" to runqueue
09-16-2016 14:11:56.653 +0000 DEBUG ExecProcessor - cmd='python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py' Added to run queue
09-16-2016 14:11:56.653 +0000 DEBUG ExecProcessor - Running: python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py on PipelineSet 0
09-16-2016 14:11:56.653 +0000 DEBUG ExecProcessor - PipelineSet 0: Created new ExecedCommandPipe for "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py", uniqueId=1283
09-16-2016 14:11:56.738 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,737 DEBUG AzureStorageTable:237 - Starting AzureStorageTable.py 344
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,738 DEBUG AzureStorageTable:369 - XML: found configuration
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:374 - XML: found stanza AzureStorageTable://LinuxsyslogVer2v0
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'access_key'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'access_key' -> 'WYIl4RC07eU+XYiie/I8pdXFxBkcwxlPgNdVw8EEGF92Di808BCk2i/rBiaEG4ygbywJzEPSisSQsg1fg5dyUQ=='
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'dateTimeColumn'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'dateTimeColumn' -> 'TIMESTAMP'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'host'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'host' -> 'hdopeussadiag1a'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'index'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'index' -> 'bitbucket'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'interval'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'interval' -> '60'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'pollingMinutes'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'pollingMinutes' -> '3600'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'sourcetype'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'sourcetype' -> 'azure:storage:table'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'storage_account'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'storage_account' -> 'hdopeussadiag1a'
09-16-2016 14:11:56.740 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,740 DEBUG AzureStorageTable:380 - XML: found param 'table_name'
09-16-2016 14:11:56.740 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,740 DEBUG AzureStorageTable:384 - XML: 'table_name' -> 'LinuxsyslogVer2v0'
09-16-2016 14:11:56.740 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,740 DEBUG AzureStorageTable:264 - dateTimeStart = '2016-09-14T14:11:56.740547'
09-16-2016 14:11:56.740 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,740 DEBUG AzureStorageTable:271 - No dateTimeStart in inputs.conf...
09-16-2016 14:11:56.741 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,741 DEBUG AzureStorageTable:290 - Marker found for table LinuxsyslogVer2v0: 2028-06-17T20:31:07.379676+00:00
09-16-2016 14:11:56.741 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,741 DEBUG AzureStorageTable:299 - Filter string: TIMESTAMP gt datetime'2028-06-17T20:31:07.379676+00:00' and TIMESTAMP lt datetime'2028-06-20T08:31:07.379676+00:00'
09-16-2016 14:11:56.744 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,744 INFO connectionpool:657 - Starting new HTTPS connection (1): hdopeussadiag1a.table.core.windows.net
09-16-2016 14:11:59.633 +0000 DEBUG ExecProcessor - cmd='python /opt/splunk/etc/apps/splunk_app_db_connect/bin/rpcstart.py' Not added to run queue
09-16-2016 14:12:00.277 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:12:00,277 DEBUG connectionpool:350 - "GET /LinuxsyslogVer2v0()?$filter=TIMESTAMP%20gt%20datetime%272028-06-17T20%3A31%3A07.379676%2B00%3A00%27%20and%20TIMESTAMP%20lt%20datetime%272028-06-20T08%3A31%3A07.379676%2B00%3A00%27 HTTP/1.1" 200 None
09-16-2016 14:12:00.278 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:12:00,278 INFO AzureStorageTable:303 - Query returned 0 results.
09-16-2016 14:12:00.278 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:12:00,278 DEBUG AzureStorageTable:336 - No results found. Checkpoiting the end date/time used for query: 2028-06-20T08:31:07.379676+00:00
09-16-2016 14:12:00.278 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:12:00,278 DEBUG AzureStorageTable:343 - Ending AzureStorageTable.py 344
09-16-2016 14:12:00.290 +0000 DEBUG ExecProcessor - PipelineSet 0: Got EOF from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py", uniqueId=1283
09-16-2016 14:12:00.300 +0000 DEBUG ExecProcessor - PipelineSet 0: Ran script: python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py, took 3.647290 seconds to run, 0 bytes read

It seems that the date ranges are wrong!?!?

[brent_weaver]

The world would be a better place if everyone was like the splunk community. THANK YOU!

[brent_weaver]

Yes you are correct. I apologize for the lack of clarity.

[aaraneta_splunk]

No problem, thanks for clarifying. I just edited your post so we can try to get more visibility for you 🙂

[aaraneta_splunk]

Hi @brent_weaver - Just to clarify your question for other users, when you say "Azure Splunk App" are you referring to the Splunk Add-on for Microsoft Azure https://splunkbase.splunk.com/app/3084/