Deployment app suddenly stopped indexing monitored file- How to troubleshoot?

[phamxuantung]

Hello,

I have an deployment app that monitor log file from an external server that work fine since last year. But suddenly, since 26/1/2023 untill now, it can't index anything. Nothing changed from the server side or on my side either, the host still produce log file on a daily basis.

I also request to check the connection and restart deployment client but no improvement.

My input.config is:

[monitor:///u01/pv/log-1/data/trafficmanager/enriched/access/*.log]
disabled = 0
index = my index
sourcetype = my sourcetype

The example log file name is: access_worker_6_2023_01_26.log 

I like to resolve this problem, even redo every step if I have to because this is urgent. And I like to know how to troubleshoot step by step to know where is the problem, and how to prevent this in the future.

[gcusello]

Hi @phamxuantung,

could you share a sample of your logs?

when does your ingestion stopped: today or the 1st of the month?

if the 1st of the month, probably the problem is the timestamp recognition, but to help you I need a sample of your logs.

Ciao.

Giuseppe

[phamxuantung]

Sorry for the late reply, this is the sample of the log, from line 1 foward:

api_key,api_method_name,bytes,cache_hit,client_transfer_time,connect_time,endpoint_name,http_method,http_status_code,http_version,oauth_access_token,package_name,package_uuid,plan_name,plan_uuid,pre_transfer_time,qps_throttle_value,quota_value,referrer,remote_total_time,request_host_name,request_id,request_time,request_uuid,response_string,service_definition_endpoint_uuid,service_id,service_name,src_ip,ssl_enabled,total_request_exec_time,traffic_manager,traffic_manager_error_code,uri,user_agent,org_name,org_uuid,sub_org_name,sub_org_uuid
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641598.598_unknown_unknown,2023-02-05T23:59:58,dafeac38-123d-4bb7-aa1c-59680afbc0b2,596 Service Not Found (Proxy),-,unknown,-,10.244.1.0,1,0.0,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641608.030_unknown_unknown,2023-02-06T00:00:08,e4cd645a-5471-4097-baf0-67f90f4d2cee,596 Service Not Found (Proxy),-,unknown,-,10.244.1.0,1,0.001,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641618.607_unknown_unknown,2023-02-06T00:00:18,ee18e506-2ea5-4792-a586-f0274e6c823b,596 Service Not Found (Proxy),-,unknown,-,10.244.1.0,1,0.0,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641627.988_unknown_unknown,2023-02-06T00:00:27,5cc9f704-61a3-443c-b670-26373afe5502,596 Service Not Found (Proxy),-,unknown,-,10.244.1.0,1,0.0,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641633.592_unknown_unknown,2023-02-06T00:00:33,8a4a97c6-9fc6-4f67-9165-a55e3cd67979,596 Service Not Found (Proxy),-,unknown,-,10.244.3.1,1,0.0,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641628.644_unknown_unknown,2023-02-06T00:00:28,251c26bb-4dfd-44b2-b88a-0143fb7148da,596 Service Not Found (Proxy),-,unknown,-,10.244.1.0,1,0.0,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641638.625_unknown_unknown,2023-02-06T00:00:38,c18cd8de-18f7-4bd8-b5bc-90d244fe32fd,596 Service Not Found (Proxy),-,unknown,-,10.244.1.0,1,0.0,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641642.970_unknown_unknown,2023-02-06T00:00:42,d71a2b1b-d438-4e5e-8173-e48f0f129d6e,596 Service Not Found (Proxy),-,unknown,-,10.244.3.1,1,0.0,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-

Curiously, the log stop after 9/1, comeback at 26/1 with only 2 line of log and then stop since that time.

The only change they made was change the name of the log from on access_worker6.log to access_worker_6_YYYY_MM_DD.log. But I in input.conf I put it as /*.log then it should catch it nonetheless.