Why is DB Connect failing?

[edwardrose]

Hello All

DB connect is failing for some weird reason. I get the following in my logs. It connects successfully but it fails to return any data. When I setup the inputs I can see data for the query but that is the only time. I have another input setup for the same exact database and it works just fine. Ideas as the logs really don't give me a direction to go.

thanks
ed

[eventdata]
connection = svr-sql-lnl-14
description = Eventdata
disabled = 0
host = svr-sql-lnl-14
index = lenel
index_time_mode = dbColumn
input_timestamp_column_number = 13
input_timestamp_format = "yyyy-MM-dd HH:mm:ss.SSZ"
interval = 30
mode = rising
query = SELECT *\
FROM "events"."dbo"."EventLog"\
WHERE UNQ_KEY > ?\
ORDER BY  UNQ_KEY ASC
source = dbo.eventviews
sourcetype = event_data
tail_rising_column_number = 1
fetch_size = 300

4/30/18 9:34:35.905 AM 2018-04-30 09:34:35.905 -0700 [QuartzScheduler_Worker-17] INFO org.easybatch.core.job.BatchJob - Job 'eventdata' finished with status: FAILED host = splk-srch-01 source
= opt/splunk/var/log/splunk/splunk_app_db_connect_server.log sourcetype = dbx_server 4/30/18 9:34:35.905 AM 2018-04-30 09:34:35.905
-0700 INFO c.s.dbx.server.task.listeners.JobMetricsListener
- action=collect_job_metrics connection=svr-sql-lnl-14 jdbc_url=null status=FAILED input_name=eventdata batch_size=1000 error_threshold=N/A is_jmx_monitoring=false start_time=2018-04-30_09:34:35 end_time=2018-04-30_09:34:35 duration=9 read_count=0 write_count=0 filtered_count=0 error_count=0 host
= splk-srch-01 source = /opt/splunk/var/log/splunk/splunk_app_db_connect_job_metrics.log sourcetype = dbx_job_metrics 4/30/18 9:34:35.898 AM 2018-04-30 09:34:35.898
-0700 [QuartzScheduler_Worker-17] INFO com.splunk.dbx.connector.logger.AuditLogger
- operation=dbinput connection_name=svr-sql-lnl-14 stanza_name=eventdata state=success sql='SELECT * FROM "events"."dbo"."EventLog" WHERE UNQ_KEY > ? ORDER BY UNQ_KEY ASC' host = splk-srch-01 source
= /opt/splunk/var/log/splunk/splunk_app_db_connect_audit_server.log sourcetype = dbx_audit 4/30/18 9:34:35.896 AM 2018-04-30 09:34:35.896
-0700 [QuartzScheduler_Worker-17] INFO c.s.d.s.dbinput.recordreader.DbInputRecordReader
- action=db_input_record_reader_is_opened input_task="eventdata" query=SELECT * FROM "events"."dbo"."EventLog" WHERE UNQ_KEY > ? ORDER BY UNQ_KEY ASC host
= splk-srch-01 source = /opt/splunk/var/log/splunk/splunk_app_db_connect_server.log sourcetype = dbx_server 4/30/18 9:34:35.896 AM 2018-04-30 09:34:35.896
-0700 [QuartzScheduler_Worker-17] INFO org.easybatch.core.job.BatchJob
- Job 'eventdata' starting host = splk-srch-01 source = /opt/splunk/var/log/splunk/splunk_app_db_connect_server.log sourcetype = dbx_server 4/30/18 9:34:05.904 AM 2018-04-30 09:34:05.904
-0700 [QuartzScheduler_Worker-14] INFO org.easybatch.core.job.BatchJob
- Job 'eventdata' finished with status: FAILED host = splk-srch-01 source
= /opt/splunk/var/log/splunk/splunk_app_db_connect_server.log sourcetype = dbx_server 4/30/18 9:34:05.904 AM 2018-04-30 09:34:05.904
-0700 INFO c.s.dbx.server.task.listeners.JobMetricsListener
- action=collect_job_metrics connection=svr-sql-lnl-14 jdbc_url=null status=FAILED input_name=eventdata batch_size=1000 error_threshold=N/A is_jmx_monitoring=false start_time=2018-04-30_09:34:05 end_time=2018-04-30_09:34:05 duration=9 read_count=0 write_count=0 filtered_count=0 error_count=0 host
= splk-srch-01 source = /opt/splunk/var/log/splunk/splunk_app_db_connect_job_metrics.log sourcetype = dbx_job_metrics 4/30/18 9:34:05.898 AM 2018-04-30 09:34:05.898
-0700 [QuartzScheduler_Worker-14] INFO com.splunk.dbx.connector.logger.AuditLogger
- operation=dbinput connection_name=svr-sql-lnl-14 stanza_name=eventdata state=success sql='SELECT * FROM "events"."dbo"."EventLog" WHERE UNQ_KEY > ? ORDER BY UNQ_KEY ASC' host = splk-srch-01 source
= /opt/splunk/var/log/splunk/splunk_app_db_connect_audit_server.log sourcetype = dbx_audit 4/30/18 9:34:05.895 AM 2018-04-30 09:34:05.895 -0700 [QuartzScheduler_Worker-14] INFO c.s.d.s.dbinput.recordreader.DbInputRecordReader
- action=db_input_record_reader_is_opened input_task="eventdata" query=SELECT * FROM "events"."dbo"."EventLog" WHERE UNQ_KEY > ? ORDER BY UNQ_KEY ASC host
= splk-srch-01 source = /opt/splunk/var/log/splunk/splunk_app_db_connect_server.log sourcetype = dbx_server 4/30/18 9:34:05.895 AM 2018-04-30 09:34:05.895
-0700 [QuartzScheduler_Worker-14] INFO org.easybatch.core.job.BatchJob

[edwardrose]

This has been resolved by a Splunk consultant that we had come onsite a while ago.

[iamarkaprabha]

cool.
That's good to hear 🙂
what was the issue ?

[iamarkaprabha]

Hi ,

Can you share the dbx logs for the particular issue

[oangarita]

10/10/18

5:30:06.546 PM

2018-10-10 17:30:06.546 +0200 [QuartzScheduler_Worker-25] INFO org.easybatch.core.job.BatchJob - Job 'sccm_updates_info' finished with status: COMPLETED

host =  MOL11101    
index = _internal   
linecount = 1   
punct = --_::._+__[-]___...._-__''___:_ 
source =    E:\Program Files\Splunk\var\log\splunk\splunk_app_db_connect_server.log 
sourcetype =    dbx_server  
splunk_server = mol18119.enagas.eng 
splunk_server_group =   dmc_group_indexer   splunk_server_group =   dmc_indexerclustergroup_ENAGAS CLUSTER  

10/10/18

5:30:06.539 PM

2018-10-10 17:30:06.539 +0200 [QuartzScheduler_Worker-25] INFO org.easybatch.core.job.BatchJob - Job 'sccm_updates_info' stopping

host =  MOL11101    
index = _internal   
linecount = 1   
punct = --_::._+__[-]___...._-__''_ 
source =    E:\Program Files\Splunk\var\log\splunk\splunk_app_db_connect_server.log 
sourcetype =    dbx_server  
splunk_server = mol18119.enagas.eng 
splunk_server_group =   dmc_group_indexer   splunk_server_group =   dmc_indexerclustergroup_ENAGAS CLUSTER  

10/10/18

5:30:03.323 PM

2018-10-10 17:30:03.323 +0200 [QuartzScheduler_Worker-25] INFO org.easybatch.core.job.BatchJob - Job 'sccm_updates_info' started

host =  MOL11101    
index = _internal   
linecount = 1   
punct = --_::._+__[-]___...._-__''_ 
source =    E:\Program Files\Splunk\var\log\splunk\splunk_app_db_connect_server.log 
sourcetype =    dbx_server  
splunk_server = mol18119.enagas.eng 
splunk_server_group =   dmc_group_indexer   splunk_server_group =   dmc_indexerclustergroup_ENAGAS CLUSTER  

10/10/18

5:30:03.181 PM

2018-10-10 17:30:03.181 +0200 [QuartzScheduler_Worker-25] INFO com.splunk.dbx.connector.logger.AuditLogger - operation=dbinput connection_name=SCCM_Produccion stanza_name=sccm_updates_info state=success sql='SELECT v_UpdateInfo.ApplicabilityCondition,
v_UpdateInfo.ApplicableAtUserLogon,
v_UpdateInfo.ArticleID,
v_UpdateInfo.BulletinID,
v_UpdateInfo.CIType_ID,
Show all 61 lines

host =  MOL11101    
index = _internal   
linecount = 61  
punct = --_::._+__[-]___....._-_=_=_=_=_='_.,_________.,__  
source =    E:\Program Files\Splunk\var\log\splunk\splunk_app_db_connect_audit_server.log   
sourcetype =    dbx_audit   
splunk_server = mol18119.enagas.eng 
splunk_server_group =   dmc_group_indexer   splunk_server_group =   dmc_indexerclustergroup_ENAGAS CLUSTER  

10/10/18

5:30:00.008 PM

2018-10-10 17:30:00.008 +0200 [QuartzScheduler_Worker-25] INFO c.s.d.s.dbinput.recordreader.DbInputRecordReader - action=db_input_record_reader_is_opened input_task="sccm_updates_info" query=SELECT v_UpdateInfo.ApplicabilityCondition,
v_UpdateInfo.ApplicableAtUserLogon,
v_UpdateInfo.ArticleID,
v_UpdateInfo.BulletinID,
v_UpdateInfo.CIType_ID,
v_UpdateInfo.CIVersion,
v_UpdateInfo.CI_CRC,
v_UpdateInfo.CI_ID,
v_UpdateInfo.CI_UniqueID,
v_UpdateInfo.ConfigurationFlags,
v_UpdateInfo.ContentSourcePath,
v_UpdateInfo.CreatedBy,
v_UpdateInfo.CustomSeverity,
v_UpdateInfo.DateCreated,
v_UpdateInfo.DateLastModified,
v_UpdateInfo.DatePosted,
v_UpdateInfo.DateRevised,
v_UpdateInfo.Description,
v_UpdateInfo.EULAAccepted,
v_UpdateInfo.EULAExists,
v_UpdateInfo.EULASignoffDate,
v_UpdateInfo.EULASignoffUser,
v_UpdateInfo.EffectiveDate,
v_UpdateInfo.InUse,
v_UpdateInfo.InfoURL,
v_UpdateInfo.IsBroken,
v_UpdateInfo.IsBundle,
v_UpdateInfo.IsChild,
v_UpdateInfo.IsDeployed,
v_UpdateInfo.IsEnabled,
v_UpdateInfo.IsExpired,
v_UpdateInfo.IsHidden,
v_UpdateInfo.IsLatest,
v_UpdateInfo.IsQuarantined,
v_UpdateInfo.IsSignificantRevision,
v_UpdateInfo.IsSuperseded,
v_UpdateInfo.IsTombstoned,
v_UpdateInfo.IsUserCI,
v_UpdateInfo.IsUserDefined,
v_UpdateInfo.LastModifiedBy,
v_UpdateInfo.LocaleID,
v_UpdateInfo.Locales,
v_UpdateInfo.MaxExecutionTime,
v_UpdateInfo.MinSourceVersion,
v_UpdateInfo.ModelId,
v_UpdateInfo.ModelName,
v_UpdateInfo.ModifiedTime,
v_UpdateInfo.PermittedUses,
v_UpdateInfo.PlatformType,
v_UpdateInfo.Precedence,
v_UpdateInfo.RequiresExclusiveHandling,
v_UpdateInfo.RevisionNumber,
v_UpdateInfo.SDMPackageDigest,
v_UpdateInfo.SDMPackageVersion,
v_UpdateInfo.SedoObjectVersion,
v_UpdateInfo.Severity,
v_UpdateInfo.SourceSite,
v_UpdateInfo.Title,
v_UpdateInfo.UpdateSource_ID,
v_UpdateInfo.UpdateType
FROM v_UpdateInfo
Collapse

host =  MOL11101    
index = _internal   
linecount = 61  
punct = --_::._+__[-]___......_-_=_=""_=_.,_________.,____  
source =    E:\Program Files\Splunk\var\log\splunk\splunk_app_db_connect_server.log 
sourcetype =    dbx_server  
splunk_server = mol18119.enagas.eng 
splunk_server_group =   dmc_group_indexer   splunk_server_group =   dmc_indexerclustergroup_ENAGAS CLUSTER  

10/10/18

5:30:00.008 PM

10/10/18

5:30:00.008 PM

2018-10-10 17:30:00.008 +0200 [QuartzScheduler_Worker-25] INFO org.easybatch.core.job.BatchJob - Job 'sccm_updates_info' starting

host =  MOL11101    
index = _internal   
linecount = 1   
punct = --_::._+__[-]___...._-__''_ 
source =    E:\Program Files\Splunk\var\log\splunk\splunk_app_db_connect_server.log 
sourcetype =    dbx_server  
splunk_server = mol18119.enagas.eng 
splunk_server_group =   dmc_group_indexer   splunk_server_group =   dmc_indexerclustergroup_ENAGAS CLUSTER  

This was after make a change in the query, because I found this mssg: 2018-10-10 16:20:03.470 +0200 [QuartzScheduler_Worker-10] WARN c.s.d.s.d.r.iterator.EventPayloadRecordIterator - input sccm_updates_info contains binary columns, will be discarded. column name:RevisionTag

So I remove the RevisionTag.

But it still not work

[oangarita]

Hi Edward,

I have the same problem. Maybe you find something and can tell us?..

Thank you,