Deployment Architecture

Deployment app suddenly stopped indexing monitored file- How to troubleshoot?

phamxuantung
Communicator

Hello,

I have an deployment app that monitor log file from an external server that work fine since last year. But suddenly, since 26/1/2023 untill now, it can't index anything. Nothing changed from the server side or on my side either, the host still produce log file on a daily basis.

I also request to check the connection and restart deployment client but no improvement.

My input.config is:

[monitor:///u01/pv/log-1/data/trafficmanager/enriched/access/*.log]
disabled = 0
index = my index
sourcetype = my sourcetype

The example log file name is: access_worker_6_2023_01_26.log 

I like to resolve this problem, even redo every step if I have to because this is urgent. And I like to know how to troubleshoot step by step to know where is the problem, and how to prevent this in the future.

 

Labels (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @phamxuantung,

could you share a sample of your logs?

when does your ingestion stopped: today or the 1st of the month?

if the 1st of the month, probably the problem is the timestamp recognition, but to help you I need a sample of your logs.

Ciao.

Giuseppe

0 Karma

phamxuantung
Communicator

Sorry for the late reply, this is the sample of the log, from line 1 foward:

api_key,api_method_name,bytes,cache_hit,client_transfer_time,connect_time,endpoint_name,http_method,http_status_code,http_version,oauth_access_token,package_name,package_uuid,plan_name,plan_uuid,pre_transfer_time,qps_throttle_value,quota_value,referrer,remote_total_time,request_host_name,request_id,request_time,request_uuid,response_string,service_definition_endpoint_uuid,service_id,service_name,src_ip,ssl_enabled,total_request_exec_time,traffic_manager,traffic_manager_error_code,uri,user_agent,org_name,org_uuid,sub_org_name,sub_org_uuid
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641598.598_unknown_unknown,2023-02-05T23:59:58,dafeac38-123d-4bb7-aa1c-59680afbc0b2,596 Service Not Found (Proxy),-,unknown,-,10.244.1.0,1,0.0,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641608.030_unknown_unknown,2023-02-06T00:00:08,e4cd645a-5471-4097-baf0-67f90f4d2cee,596 Service Not Found (Proxy),-,unknown,-,10.244.1.0,1,0.001,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641618.607_unknown_unknown,2023-02-06T00:00:18,ee18e506-2ea5-4792-a586-f0274e6c823b,596 Service Not Found (Proxy),-,unknown,-,10.244.1.0,1,0.0,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641627.988_unknown_unknown,2023-02-06T00:00:27,5cc9f704-61a3-443c-b670-26373afe5502,596 Service Not Found (Proxy),-,unknown,-,10.244.1.0,1,0.0,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641633.592_unknown_unknown,2023-02-06T00:00:33,8a4a97c6-9fc6-4f67-9165-a55e3cd67979,596 Service Not Found (Proxy),-,unknown,-,10.244.3.1,1,0.0,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641628.644_unknown_unknown,2023-02-06T00:00:28,251c26bb-4dfd-44b2-b88a-0143fb7148da,596 Service Not Found (Proxy),-,unknown,-,10.244.1.0,1,0.0,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641638.625_unknown_unknown,2023-02-06T00:00:38,c18cd8de-18f7-4bd8-b5bc-90d244fe32fd,596 Service Not Found (Proxy),-,unknown,-,10.244.1.0,1,0.0,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-
unknown,-,30,0,0.0,0.0,-,POST,596,HTTP/1.1,-,-,-,-,-,0.0,0,0,-,0.0,developer.napas.com.vn,1675641642.970_unknown_unknown,2023-02-06T00:00:42,d71a2b1b-d438-4e5e-8173-e48f0f129d6e,596 Service Not Found (Proxy),-,unknown,-,10.244.3.1,1,0.0,tm-deploy-0-97674db57-smcdv,ERR_596_SERVICE_NOT_FOUND,/healthcheck,-,-,-,-,-

Curiously, the log stop after 9/1, comeback at 26/1 with only 2 line of log and then stop since that time.

log.PNG

The only change they made was change the name of the log from on access_worker6.log to access_worker_6_YYYY_MM_DD.log. But I in input.conf I put it as /*.log then it should catch it nonetheless.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...