Hi @jkat54 , We configure above query in inputs.conf file and able recovery few data from april 9th to april 14th but it's only pull very first time and then it's not pulling at-all. Is there any configuration need to update or KV store side update ? log_analytics_query = ContainerLog | where _ResourceId != "" | where TimeGenerated between(datetime("2020-04-09 00:00:00") .. datetime("2020-04-14 23:59:59")) start_date = 01/01/1970 00:00:00   ... View more

Hi @jkat54 , Can you please let me know above API query could use in this add-on to pull this 7 days backlogs data as direct query is resulting? ... View more

Hi @jkat54 , Many thanks for your above all information & appreciated support. We are trying to update your add-on but we need your help, is it could be possible to fetch back data by inputs.conf file 's log_analytics_query with time range as below? as by using this query i can getch backlog april 2020 data by direct API call. log_analytics_query=AzureDiagnostics | where ResourceGroup != "" | where TimeGenerated between(datetime("2020-04-09 00:00:00")..datetime("2020-04-14 23:59:59"))  Also please suggest us inside your add-on which python files is causing this data pull based on start_date?  So that we could also debug on these codes? Are there any Splunk PS support as it;s quite urgent?   ... View more

Hi @jkat54 , Do you feel anything problem? Why once i used start_date as 0 it's giving ValueError: time data '0' does not match format '%d/%m/%Y %H:%M:%S'. but if i m giving some date it is working to getting current data<=15 mints data but not past 4 months data props.conf [azure:loganalytics:ad*] DATETIME_CONFIG = INDEXED_EXTRACTIONS = json NO_BINARY_CHECK = true TIMESTAMP_FIELDS = XYZ TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%Z disabled = false TZ = Europe/Stockholm TIME_PREFIX = category = Structured KV_MODE = SHOULD_LINEMERGE = true   inputs.conf [log_analytics://GFDG_HSE_AD_YYYYYYY] application_id = TYU application_key = XXXXXXXXXXXXX event_delay_lag_time = 900 index = jh_test sourcetype = azure:loganalytics:ad:AuditLogs interval = 300 log_analytics_query = ContainerLog | where _ResourceId != "" resource_group = njjj-gff-oug-gh start_date = 0 subscription_id = MNO tenant_id = KPL workspace_id = HFR disabled = 0   ... View more

2020-07-15 09:08:43,477 INFO pid=124798 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1 2020-07-15 09:08:44,494 INFO pid=124798 tid=MainThread file=setup_util.py:log_info:114 | Log level is not set, use default INFO 2020-07-15 09:08:44,494 INFO pid=124798 tid=MainThread file=splunk_rest_client.py:_request_handler:100 | Use HTTP connection pooling 2020-07-15 09:08:44,495 INFO pid=124798 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1 2020-07-15 09:08:44,504 ERROR pid=124798 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events. Traceback (most recent call last): File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/modinput_wrapper/base_modinput.py", line 127, in stream_events self.collect_events(ew) File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py", line 96, in collect_events input_module.collect_events(self, ew) File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/input_module_log_analytics.py", line 39, in collect_events start_datetime = datetime.datetime.strptime(str(input_item['start_date']),'%d/%m/%Y %H:%M:%S') File "/opt/splunk_temp/splunk/lib/python2.7/_strptime.py", line 332, in _strptime (data_string, format)) ValueError: time data '0' does not match format '%d/%m/%Y %H:%M:%S' ... View more

Hi @jkat54 , Just more clarifying on my requirement here that i wanted pull my backlog data from April 9th 2020 to April 14th 2020 but this add-on is not able to get past back logs data. Could you please help me how could i pull these 7 old days data? Yes, we can see it's pull some back days data one time on very first time run this app but after that it's only pulling current data not back days data.     ... View more

Thanks , Seem to be i am now getting below error like ERRORtime data '0' does not match format '%d/%m/%Y %H:%M:%S' HFR01   2020-07-14 19:25:29.052 message from "python /opt/prd_sec/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" ERRORtime data '0' does not match format '%d/%m/%Y %H:%M:%S' HFR01 2020-07-14 19:25:29.021 message from "python /opt/prd_sec/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" ValueError: time data '0' does not match format '%d/%m/%Y %H:%M:%S' HFR01 2020-07-14 19:25:29.021 message from "python /opt/prd_sec/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" (data_string, format)) HFR01 2020-07-14 19:25:29.021 message from "python /opt/prd_sec/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/prd_sec/splunk/lib/python2.7/_strptime.py", line 332, in _strptime HFR01 2020-07-14 19:25:29.021 message from "python /opt/prd_sec/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" start_datetime = datetime.datetime.strptime(str(input_item['start_date']),'%d/%m/%Y %H:%M:%S') HFR01 2020-07-14 19:25:29.021 message from "python /opt/prd_sec/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/prd_sec/splunk/etc/apps/TA-ms-loganalytics/bin/input_module_log_analytics.py", line 39, in collect_events HFR01 2020-07-14 19:25:29.021 message from "python /opt/prd_sec/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" input_module.collect_events(self, ew) HFR01 2020-07-14 19:25:29.021 message from "python /opt/prd_sec/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/prd_sec/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py", line 96, in collect_events HFR01 2020-07-14 19:25:29.021 message from "python /opt/prd_sec/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" self.collect_events(ew) HFR01 2020-07-14 19:25:29.021 message from "python /opt/prd_sec/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/prd_sec/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/modinput_wrapper/base_modinput.py", line 127, in stream_events HFR01 2020-07-14 19:25:29.021 message from "python /opt/prd_sec/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" Traceback (most recent call last):   ... View more

Hi @jkat54,  Could you please suggest me on my above question? We can see some data (at-least few data from past 30 days only) not 100% backlog data from past 150 days. Even if same scenario if lag as 900 seconds & interval as 60 or 300 seconds in all scenario just pulling few days data even if my under-layer OMS/Storage log analytics workspace is having enough back log data for past 150 days. Then why it is not pulling backlogs data 100% but its pulling new future data fine. My requirement here is at-least i could pull or recovery my all old days data (past 150 days) backlog data 100%. I am waiting your appreciated help here or please suggest any codes need to update here? Please see attached graph here & very first graph is yesterday count & second one is today count? Seem to be backlog count is not increases even if storage workspace data are available for past 150 days.         ... View more

Hi @jkat54,  Here is a scenario is that my OMS/Log Analytics has been stored data past 150 days i we just configure lag is 900 seconds =15 minutes and Interval = 300 seconds = 5 mints but i can noticed that data has been pulling but it has been huge delay. Also it has been pulling only past 1 months data but not pulling data for past 5 months. Hence i took lag as 360 seconds = 6 minutes and interval is 180 minutes= 3 minutes i can see little fast data pulling but it pulling past 30 days only & pulling is in progress. Still we have data available past 5 months and we have Startdate as i put exactly before 5 months. I don't know it will pull 5 months data or not?  As i can see only it's pulling data from current time to past 1 month and current time to future time. Could you please guide me how could i pull the data only for past 5 months?     ... View more

Thanks @jkat54  ... View more

Excellent discussion and your appreciated assistance @jkat54. Could you share me url more details configuration on inputs.conf (startdate) and checkpoint file. I wanted to try recovery some backlogs data by using this second instance now both are working. But i wanted second instance LA should configure such way so that i could recover my backlogs data example i wanted recovery my april 2020 15th  april to 20th april data from LA Workspace from Azure side. While i am validating data its first pulling current data and then go-ahead to some backward but not pulling all backlogs data. Can i customizing some backlogs time duration pulling data?     ... View more

Thanks mate... ... View more

This has been successfully Fixed. Kv store port issue - Fixed Password issue -Fixed Datetime /start time issue -Fixed error_description":"AADSTS7000215: Invalid client secret is provided – Fixed Secret key issue has been fixed Password issue has been fixed ... View more

Hi, After fixing all errors then getting below    2020-07-07 03:06:25.203 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" ERRORGet Token request returned http error: 401 and server response: {"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret is provided.\r\nTrace ID: b6e89b21-ad56-42fe-a006-3177419a8600\r\nCorrelation ID: 68a7a5b7-4acb-4fdc-abf7-5a77edce87b3\r\nTimestamp: 2020-07-07 01:06:25Z","error_codes":[7000215],"timestamp":"2020-07-07 01:06:25Z","trace_id":"b6e89b21-ad56-42fe-a006-3177419a8600","correlation_id":"68a7a5b7-4acb-4fdc-abf7-5a77edce87b3","error_uri":"https://login.microsoftonline.com/error?code=7000215"} HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" AdalError: Get Token request returned http error: 401 and server response: {"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret is provided.\r\nTrace ID: b6e89b21-ad56-42fe-a006-3177419a8600\r\nCorrelation ID: 68a7a5b7-4acb-4fdc-abf7-5a77edce87b3\r\nTimestamp: 2020-07-07 01:06:25Z","error_codes":[7000215],"timestamp":"2020-07-07 01:06:25Z","trace_id":"b6e89b21-ad56-42fe-a006-3177419a8600","correlation_id":"68a7a5b7-4acb-4fdc-abf7-5a77edce87b3","error_uri":"https://login.microsoftonline.com/error?code=7000215"} HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" raise AdalError(return_error_string, error_response) HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/adal/oauth2_client.py", line 281, in get_token HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" return client.get_token(oauth_parameters) HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/adal/token_request.py", line 113, in _oauth_get_token HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" token = self._oauth_get_token(oauth_parameters) HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/adal/token_request.py", line 316, in get_token_with_client_credentials HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" return token_request.get_token_with_client_credentials(client_secret) HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/adal/authentication_context.py", line 158, in token_func HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" return token_func(self) HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/adal/authentication_context.py", line 109, in _acquire_token HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" return self._acquire_token(token_func) HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/adal/authentication_context.py", line 160, in acquire_token_with_client_credentials HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" token_response = context.acquire_token_with_client_credentials('https://api.loganalytics.io/', application_id, application_key) HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/input_module_log_analytics.py", line 49, in collect_events HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" input_module.collect_events(self, ew) HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py", line 96, in collect_events HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" self.collect_events(ew) HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/modinput_wrapper/base_modinput.py", line 127, in stream_events HFR01 2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" Traceback (most recent call last): HFR01 2020-07-07 03:06:23.846 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" ERRORGet Token request returned http error: 401 and server response: {"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret is provided.\r\nTrace ID: bf76db00-dd5d-46fa-a8e7-789d40cf7c00\r\nCorrelation ID: f707f17a-874b-420c-943f-0131e62a565c\r\nTimestamp: 2020-07-07 01:06:23Z","error_codes":[7000215],"timestamp":"2020-07-07 01:06:23Z","trace_id":"bf76db00-dd5d-46fa-a8e7-789d40cf7c00","correlation_id":"f707f17a-874b-420c-943f-0131e62a565c","error_uri":"https://login.microsoftonline.com/error?code=7000215"} HFR01 2020-07-07 03:06:23.821 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" AdalError: Get Token request returned http error: 401 and server response: {"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret is provided.\r\nTrace ID: bf76db00-dd5d-46fa-a8e7-789d40cf7c00\r\nCorrelation ID: f707f17a-874b-420c-943f-0131e62a565c\r\nTimestamp: 2020-07-07 01:06:23Z","error_codes":[7000215],"timestamp":"2020-07-07 01:06:23Z","trace_id":"bf76db00-dd5d-46fa-a8e7-789d40cf7c00","correlation_id":"f707f17a-874b-420c-943f-0131e62a565c","error_uri":"https://login.microsoftonline.com/error?code=7000215"} HFR01 2020-07-07 03:06:20.190 New scheduled exec process: python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py HFR01 2020-07-07 03:06:19.513 Found external scheme definition for stanza="log_analytics://" from spec file="/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/README/inputs.conf.spec" with parameters="resource_group, workspace_id, subscription_id, tenant_id, application_id, application_key, log_analytics_query, start_date, event_delay_lag_time"   ... View more

For first instance i am getting below error but still able to see data flow. 2020-07-07 01:35:59.377 message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" Modular input azure_diagnostic_logs://Azure Monitor Diagnostic Logs _ GOEP DEV Error getting event hub creds: Error: Get Token request returned http error: 401 and server response: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials\r\n ... View more

Thanks Mate, Checking kvstore port [8192]: open Restart time i am able to get this message. the same password.conf file whichone is working in my first instance same one i am using second instance hence trying to fixing this issue.   ... View more

Actually i have different port for two instance. First instance is having 8191 kvstore  port & this is working. But second one is having 8192 port is havinf error.       2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" HTTPError: HTTP 500 Internal Server Error -- {"messages":[{"type":"ERROR","text":"Unexpected error \"<class 'splunktaucclib.rest_handler.error.RestError'>\" from python handler: \"REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File \"/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/handler.py\", line 113, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File \"/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/handler.py\", line 348, in _format_all_response\n self._encrypt_raw_credentials(cont['entry'])\n File \"/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/handler.py\", line 382, in _encrypt_raw_credentials\n change_list = rest_credentials.decrypt_all(data)\n File \"/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/credentials.py\", line 286, in decrypt_all\n all_passwords = credential_manager._get_all_passwords()\n File \"/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/utils.py\", line 154, in wrapper\n return func(*args, **kwargs)\n File \"/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/credentials.py\", line 272, in _get_all_passwords\n clear_password += field_clear[index]\nTypeError: cannot concatenate 'str' and 'NoneType' objects\n\". See splunkd.log for more details."}]}HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" raise HTTPError(response) HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/packages/splunklib/binding.py", line 1221, in request HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" return self.request(url, { 'method': "GET", 'headers': headers }) HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/packages/splunklib/binding.py", line 1160, in get HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" response = self.http.get(path, self._auth_headers, **query) HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/packages/splunklib/binding.py", line 665, in get HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" val = f(*args, **kwargs) HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/packages/splunklib/binding.py", line 69, in new_f HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" return request_fun(self, *args, **kwargs) HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/packages/splunklib/binding.py", line 287, in wrapper HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" **query HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/global_config/configuration.py", line 175, in _load_endpoint HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" input_item['entity'] HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/global_config/configuration.py", line 270, in load HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" ucc_inputs = global_config.inputs.load(input_type=self.input_type) HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/modinput_wrapper/base_modinput.py", line 171, in _parse_input_args_from_global_config HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" self._parse_input_args_from_global_config(inputs) HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/modinput_wrapper/base_modinput.py", line 152, in parse_input_args HFR01 2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" self.parse_input_args(input_definition) ... View more

First instance - 8191 & second instance - 8192 both are different port and getting this http 500 & this credential issue. Do you suggest then next step?   ... View more

Yes Yes, i had taken new port for kv store. so do i need to revert to like first instance? ... View more

My bad, yes seem to be i have taken new port in second instance. So can i update this port to like first working instance? ... View more

Same conf. is working /opt/splunk/bin but why not under /opt/splunk_temp/splunk/bin location's HFR secondary instance now?  ... View more

Also both side  ./splunk show kvstore-status Your session is invalid. Please login. Splunk username: Password: This member: jvjhvhjvjvjvjvjhvhj : Ready date : Tue Jul 7 00:27:49 2020 dateSec : 1594074469.921 disabled : 0 guid : mmmmm-kkkk-hhhh-jjjjj-fffffffffffff oplogEndTimestamp : Tue Jul 7 00:27:41 2020 oplogEndTimestampSec : 1594074461 oplogStartTimestamp : Wed Jul 1 12:42:33 2020 oplogStartTimestampSec : 1593600153 port : 8192 replicaSet : vcvccvc-hgcgcjc-vncvcjgc-cgcjcgcc replicationStatus : KV store captain standalone : 1 status : ready KV store members: xxx.x.x.x:yyyy configVersion : 1 electionDate : Tue Jul 7 00:01:39 2020 electionDateSec : 1594072899 hostAndPort : xxx.x.x.x:yyyy optimeDate : Tue Jul 7 00:27:41 2020 optimeDateSec : 1594074461 replicationStatus : KV store captain uptime : 1570 ... View more

Thanks Mate, We have in a single HFR two Splunk heavy forwarder location 1. /opt/splunk/bin Here this add-on is working fine 2. /opt/splunk_temp/splunk But same configuration it is not working here with same KV store configuration & password.conf file           ... View more

@jkat54    We are also facing same issue unable to get data by this add-on having http 500 Internal server error. Please assist is asap. ... View more