All Apps and Add-ons

Splunk Add-on for Microsoft Azure: When configuring storage account inputs, do I write out the full URL or the hostname?

brent_weaver
Builder

I am having marginal success. We are writing out Linux VMs syslogs to table storage, which I can see with Azure Storage explorer but does not show up in Splunk after having added this table to storage account inputs. For storage account, do I put in the full URL or just the hostname?

Does anyone have experience with this?

0 Karma
1 Solution

jconger
Splunk Employee
Splunk Employee

The logic for the start date on the input works like this:

  • Set a start date to 2 days ago. You'll see this in the debug message "AzureStorageTable:264 - dateTimeStart = '2016-09-14T14:11:56.740547'"
  • Look for start date in inputs.conf. If we find one, use it. Otherwise use the 2 days ago one. It looks like you don't have a start date specified, so you get the debug message "No dateTimeStart in inputs.conf..."
  • Look for a marker from the check point directory (which holds the last date/time we saw when looping through table rows). If there is a marker, then use the date/time from the marker. Your data has a marker as evidenced by this message "DEBUG AzureStorageTable:290 - Marker found for table LinuxsyslogVer2v0: 2028-06-17T20:31:07.379676+00:00"

When looping through the table rows, we look at the field specified for the timestamp to see if it is greater than any other row (basically keeping a variable with the largest date/time seen in the data). After looping through, we write this value to the checkpoint to use next time. So, it looks like your data has an invalid time stamp in Azure (2028-06-17T20:31:07.379676+00:00).

You can clear the checkpoint, but it will be beneficial to use the Azure Storage Explorer to see where this errant data is logged.

View solution in original post

brent_weaver
Builder

I used this procedure to resolve this issue. It seems that somehow the input got checkpointed into 2021?!?!

Remove checkpoints
You can remove checkpoints by running the Splunk clean utility.

Caution: Be careful when removing checkpoints. Running the clean command removes your indexed data. For example, clean all removes ALL your indexed data.

For example, to remove checkpoints for a specific scheme:

splunk clean inputdata [<scheme>]

So to resolve this I ran:

splunk clean inputdata AzureTableStorage

This removed the check point file and splunk was in turn able to reset the checkpoint and carry on.

Thank you!

jconger
Splunk Employee
Splunk Employee

The logic for the start date on the input works like this:

  • Set a start date to 2 days ago. You'll see this in the debug message "AzureStorageTable:264 - dateTimeStart = '2016-09-14T14:11:56.740547'"
  • Look for start date in inputs.conf. If we find one, use it. Otherwise use the 2 days ago one. It looks like you don't have a start date specified, so you get the debug message "No dateTimeStart in inputs.conf..."
  • Look for a marker from the check point directory (which holds the last date/time we saw when looping through table rows). If there is a marker, then use the date/time from the marker. Your data has a marker as evidenced by this message "DEBUG AzureStorageTable:290 - Marker found for table LinuxsyslogVer2v0: 2028-06-17T20:31:07.379676+00:00"

When looping through the table rows, we look at the field specified for the timestamp to see if it is greater than any other row (basically keeping a variable with the largest date/time seen in the data). After looping through, we write this value to the checkpoint to use next time. So, it looks like your data has an invalid time stamp in Azure (2028-06-17T20:31:07.379676+00:00).

You can clear the checkpoint, but it will be beneficial to use the Azure Storage Explorer to see where this errant data is logged.

brent_weaver
Builder

I think this has been resolved by clearing the checkpoint file. I am doing more testing but it does seem to be getting logs. I will conclude this post with what I did to fix it if it does in fact work.

THANK YOU!

0 Karma

brent_weaver
Builder

THANK YOU so much for the great information!

I have already verified that the datetime in azure is correct.
No date in inputs.conf

Where is the checkpoint file? I cannot seem to find it even after having looked in the python script. Can this value be reset?

0 Karma

brent_weaver
Builder

The issue seems to be that the date ranges are being picked as 2021-xx-xx?!?

If I look at the log file I see this:

09-16-2016 14:11:56.741 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,741 DEBUG AzureStorageTable:299 - Filter string: ***TIMESTAMP gt datetime'2028-06-17***T20:31:07.379676+00:00' and TIMESTAMP lt datetime'2028-06-20T08:31:07.379676+00:00'

Why is this happening? ENabled debug in the python script:

/opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py

Any advice is MUCH appreciated!

0 Karma

brent_weaver
Builder

More information:

Here is from the log file:

09-16-2016 14:11:56.653 +0000 DEBUG ExecProcessor - ExecProcessorSharedState::addToRunQueue() path='python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py' restartTimerIfNeeded=1
09-16-2016 14:11:56.653 +0000 DEBUG ExecProcessor - adding "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" to runqueue
09-16-2016 14:11:56.653 +0000 DEBUG ExecProcessor - cmd='python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py' Added to run queue
09-16-2016 14:11:56.653 +0000 DEBUG ExecProcessor - Running: python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py on PipelineSet 0
09-16-2016 14:11:56.653 +0000 DEBUG ExecProcessor - PipelineSet 0: Created new ExecedCommandPipe for "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py", uniqueId=1283
09-16-2016 14:11:56.738 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,737 DEBUG AzureStorageTable:237 - Starting AzureStorageTable.py 344
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,738 DEBUG AzureStorageTable:369 - XML: found configuration
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:374 - XML: found stanza AzureStorageTable://LinuxsyslogVer2v0
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'access_key'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'access_key' -> 'WYIl4RC07eU+XYiie/I8pdXFxBkcwxlPgNdVw8EEGF92Di808BCk2i/rBiaEG4ygbywJzEPSisSQsg1fg5dyUQ=='
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'dateTimeColumn'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'dateTimeColumn' -> 'TIMESTAMP'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'host'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'host' -> 'hdopeussadiag1a'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'index'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'index' -> 'bitbucket'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'interval'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'interval' -> '60'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'pollingMinutes'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'pollingMinutes' -> '3600'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'sourcetype'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'sourcetype' -> 'azure:storage:table'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:380 - XML: found param 'storage_account'
09-16-2016 14:11:56.739 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,739 DEBUG AzureStorageTable:384 - XML: 'storage_account' -> 'hdopeussadiag1a'
09-16-2016 14:11:56.740 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,740 DEBUG AzureStorageTable:380 - XML: found param 'table_name'
09-16-2016 14:11:56.740 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,740 DEBUG AzureStorageTable:384 - XML: 'table_name' -> 'LinuxsyslogVer2v0'
09-16-2016 14:11:56.740 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,740 DEBUG AzureStorageTable:264 - dateTimeStart = '2016-09-14T14:11:56.740547'
09-16-2016 14:11:56.740 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,740 DEBUG AzureStorageTable:271 - No dateTimeStart in inputs.conf...
09-16-2016 14:11:56.741 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,741 DEBUG AzureStorageTable:290 - Marker found for table LinuxsyslogVer2v0: 2028-06-17T20:31:07.379676+00:00
09-16-2016 14:11:56.741 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,741 DEBUG AzureStorageTable:299 - Filter string: TIMESTAMP gt datetime'2028-06-17T20:31:07.379676+00:00' and TIMESTAMP lt datetime'2028-06-20T08:31:07.379676+00:00'
09-16-2016 14:11:56.744 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:11:56,744 INFO connectionpool:657 - Starting new HTTPS connection (1): hdopeussadiag1a.table.core.windows.net
09-16-2016 14:11:59.633 +0000 DEBUG ExecProcessor - cmd='python /opt/splunk/etc/apps/splunk_app_db_connect/bin/rpcstart.py' Not added to run queue
09-16-2016 14:12:00.277 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:12:00,277 DEBUG connectionpool:350 - "GET /LinuxsyslogVer2v0()?$filter=TIMESTAMP%20gt%20datetime%272028-06-17T20%3A31%3A07.379676%2B00%3A00%27%20and%20TIMESTAMP%20lt%20datetime%272028-06-20T08%3A31%3A07.379676%2B00%3A00%27 HTTP/1.1" 200 None
09-16-2016 14:12:00.278 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:12:00,278 INFO AzureStorageTable:303 - Query returned 0 results.
09-16-2016 14:12:00.278 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:12:00,278 DEBUG AzureStorageTable:336 - No results found. Checkpoiting the end date/time used for query: 2028-06-20T08:31:07.379676+00:00
09-16-2016 14:12:00.278 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py" 2016-09-16 14:12:00,278 DEBUG AzureStorageTable:343 - Ending AzureStorageTable.py 344
09-16-2016 14:12:00.290 +0000 DEBUG ExecProcessor - PipelineSet 0: Got EOF from "python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py", uniqueId=1283
09-16-2016 14:12:00.300 +0000 DEBUG ExecProcessor - PipelineSet 0: Ran script: python /opt/splunk/etc/apps/TA-Azure/bin/AzureStorageTable.py, took 3.647290 seconds to run, 0 bytes read

It seems that the date ranges are wrong!?!?

0 Karma

brent_weaver
Builder

The world would be a better place if everyone was like the splunk community. THANK YOU!

brent_weaver
Builder

Yes you are correct. I apologize for the lack of clarity.

0 Karma

aaraneta_splunk
Splunk Employee
Splunk Employee

No problem, thanks for clarifying. I just edited your post so we can try to get more visibility for you 🙂

aaraneta_splunk
Splunk Employee
Splunk Employee

Hi @brent_weaver - Just to clarify your question for other users, when you say "Azure Splunk App" are you referring to the Splunk Add-on for Microsoft Azure https://splunkbase.splunk.com/app/3084/

Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...