Splunk DB Connect App: How to troubleshoot why db input fails to fetch results sometimes?

[sudosplunk]

Hi, I am trying to troubleshoot why db input fails to fetch results sometimes.

We have a db connect input which runs every hour and it fails occasionally. When looking at splunk internal logs, I have seen two kinds of messages in in splunk_app_db_connect_job_metrics.log file. What is the meaning of read_count and write_count fields

SUCCESS MESSAGE:

2020-01-25 07:39:00.683 -0500 INFO  c.s.dbx.server.task.listeners.JobMetricsListener - action=collect_job_metrics connection=Vertica_Prod_LongRun_Conn jdbc jdbc_url=null db_read_time=0 hec_upload_time=3 hec_record_process_time=0 format_hec_success_count=1 hec_upload_bytes=647 status=COMPLETED input_name=DWH.Vertica.UnmatchedImpressionsMonitoring batch_size=1000 error_threshold=N/A is_jmx_monitoring=false start_time=2020-01-25_07:16:00 end_time=2020-01-25_07:39:00 duration=1380682 read_count=19 write_count=19 filtered_count=0 error_count=0

FAILED MESSAGE:

2020-01-25 06:27:45.809 -0500 INFO  c.s.dbx.server.task.listeners.JobMetricsListener - action=collect_job_metrics connection=Vertica_Prod_LongRun_Conn jdbc_url=null status=FAILED input_name=DWH.Vertica.UnmatchedImpressionsMonitoring batch_size=1000 error_threshold=N/A is_jmx_monitoring=false start_time=2020-01-25_06:16:00 end_time=2020-01-25_06:27:45 duration=705808 read_count=18 write_count=18 filtered_count=0 error_count=0

I attached a screenshot of logs from when query starts until it fails. Please let me know if more details are needed. Any insights into this are much appreciated!

[spl10]

Hello splunkers

I am also facing the same issue and no data collection is happening due to this issue. Can anyone please guide?

[spl10]

Hi @sudosplunk 

Is this being sorted? Could you assist me as well?

[brentdsmith]

read_count is the number of row read by the input and write is the number of rows written to an index.

[cyvi01]

What you are saying now is impossible...

2020-08-05 15:59:47.101 +0400 INFO c.s.dbx.server.task.listeners.JobMetricsListener - action=collect_job_metrics connection=XXX jdbc_url=null db_read_time=86 hec_upload_time=124 hec_record_process_time=195 format_hec_success_count=10380 hec_upload_bytes=9885449 status=COMPLETED input_name=XXXinput batch_size=1000 error_threshold=N/A is_jmx_monitoring=false start_time=2020-08-05_03:59:46 end_time=2020-08-05_03:59:47 duration=414 read_count=331755987 write_count=331599055 filtered_count=0 error_count=0

batch_size is 1000... How can i read 331755987 and write 331599055...

I´m not able to find the number of rows being read and written to the index apart from counting directly from the index and sourcetype...

[haripriyasarve1]

Hi There,

I also facing the same issue. If you have found the solution, please post it here, so it will be useful to all of us.

Thank you

[cyvi01]

I suggest you to upgrade the app first to the latest available version for your Splunk environment.