Splunk DB Connect 2 suddenly stops to retrieve events

[AlexeyNL]

Please help me to understand why my db input stops to retrieve events from Oracle dba_audit_trail system view.

Here is a piece of Splunk DB Connect 2 logs with last execution of the db input trail iterator ( the search command is
index=_internal (source=*dbx2.log OR source=*rpc.log OR source=*health.log)

and so on...
2016-07-18 20:01:45 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18T20:01:35+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:01:35 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18T20:01:25+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:01:25 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18T20:01:15+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:01:15 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18T20:01:05+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:01:05 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18T20:00:55+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:00:55 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18T20:00:45+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:00:45 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18T20:00:35+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:00:35 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18 20:00:25 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18 20:00:25 DEBUG ResultSetIterator:153 - action=result_set_metadata column_count=29
2016-07-18T20:00:25+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:00:25 INFO  HealthLogger:193 - DB_JDBC_URL=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))) FREE_MEMORY=283981504 FUNCTION=dbinputTailIterator LABEL=JP MAX_MEMORY=3162177536 PROTOCOL=HTTP SQL="SELECT * FROM (SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF')) t WHERE \"TIMESTAMP\" > ? ORDER By \"TIMESTAMP\" ASC" STATE=completed TOTAL_MEMORY=329232384 TRANS_OBJECT_ID=7e231619-ba98-4f7f-b043-d74cccc0d34c UPTIME=33474978 task_id=7f15a1e3-11c3-4124-8fbc-889a583d684a
2016-07-18 20:00:25 DEBUG DefaultDBX2Input:30 -     duration=9  splunk-user=admin   connection=oracledb_connection  rows=50000  params=[1468861158000]  sql=SELECT * FROM (SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF')) t WHERE "TIMESTAMP" > ? ORDER By "TIMESTAMP" ASC
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 542: length_of_data_sent=1593
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 523: -----------------------
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 511: Upgrade: WebSocket
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 511: Sec-WebSocket-Accept: OqtiOHz9PTRolwxclcsPwZSKT10=
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 511: Connection: Upgrade
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 511: Date: Mon, 18 Jul 2016 17:00:15 GMT
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 511: HTTP/1.1 101 Switching Protocols
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 504: --- response header ---
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 469: -----------------------
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 468: GET /ws HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: localhost:9999
Origin: http://localhost:9999
Sec-WebSocket-Key: pjyTZmA6TXyX4GAQ9N3NWw==
Sec-WebSocket-Version: 13
Collapse
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 467: --- request header ---
2016-07-18T20:00:15+0300 [DEBUG] [mi_input.py], line 52 : action=run_tail_input dbinput="mi_input://auditdb_events"
2016-07-18T20:00:15+0300 [DEBUG] [mi_input.py], line 94 : action=get_rising_column_checkpoint_for_input dbinput="mi_input://auditdb_events" checkpoint=1468861158000
2016-07-18T20:00:15+0300 [DEBUG] [mi_input.py], line 106: action=get_input_websocket websocket_url=ws_service://localhost:9999/ws
2016-07-18T20:00:15+0300 [DEBUG] [mi_input.py], line 117: action=get_enable_query_wrapping_for_connection connection=oracledb_connection enable_query_wrapping=True
2016-07-18T20:00:15+0300 [INFO] [mi_input.py], line 136: action=start_executing_dbinput dbinput="mi_input://auditdb_events"
2016-07-18T20:00:15+0300 [DEBUG] [health_logger.py], line 132: health logger is ON
2016-07-18T20:00:15+0300 [DEBUG] [health_logger.py], line 108: action=check_health_logger_config user=admin, role=admin
2016-07-18 20:00:15 INFO  HealthLogger:193 - DB_JDBC_URL=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))) FREE_MEMORY=284249248 FUNCTION=dbinputTailIterator LABEL=JP MAX_MEMORY=3162177536 PROTOCOL=HTTP SQL="SELECT * FROM (SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF')) t WHERE \"TIMESTAMP\" > ? ORDER By \"TIMESTAMP\" ASC" STATE=started TOTAL_MEMORY=329232384 TRANS_OBJECT_ID=7e231619-ba98-4f7f-b043-d74cccc0d34c UPTIME=33465008 task_id=7f15a1e3-11c3-4124-8fbc-889a583d684a
2016-07-18 20:00:15 INFO  HealthLogger:193 - FREE_MEMORY=284519280 FUNCTION=getServerStatus LABEL=JP MAX_MEMORY=3162177536 PROTOCOL=HTTP STATE=completed TOTAL_MEMORY=329232384 UPTIME=33464802 task_id=ce79b1f1-1a3e-4271-a71b-e81f3bdb13e7
2016-07-18 20:00:15 INFO  HealthLogger:193 - FREE_MEMORY=284519280 FUNCTION=getServerStatus LABEL=JP MAX_MEMORY=3162177536 PROTOCOL=HTTP STATE=started TOTAL_MEMORY=329232384 UPTIME=33464801 task_id=ce79b1f1-1a3e-4271-a71b-e81f3bdb13e7
2016-07-18 20:00:15 INFO  HealthLogger:193 - FREE_MEMORY=284760256 FUNCTION=getServerStatus LABEL=JP MAX_MEMORY=3162177536 PROTOCOL=HTTP STATE=completed TOTAL_MEMORY=329232384 UPTIME=33464775 task_id=ec8c5ddd-2def-449f-bfec-67916aa05ea9
2016-07-18 20:00:15 DEBUG Connector:183 - datasource=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))):jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))):TECH_ARCSIGHT:1651651705 driver=oracle.jdbc.OracleDriver
2016-07-18 20:00:15 DEBUG Connector:183 - datasource=oracledb_connection:jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))):TECH_ARCSIGHT:-1993644591 driver=oracle.jdbc.OracleDriver
2016-07-18 20:00:15 DEBUG Connector:180 - data_source_size=2
2016-07-18 20:00:15 DEBUG DefaultDBX2Input:316 - action=run_tail_input_iterator query="SELECT * FROM (SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF')) t WHERE "TIMESTAMP" > ? ORDER By "TIMESTAMP" ASC" checkpoint=1468861158000 rising_column=TIMESTAMP rising_column_no=1
2016-07-18 20:00:15 DEBUG DBX2Proxy:495 - action=dbinput_tail_iterator_get_query_from_client query="SELECT * FROM (SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF')) t WHERE "TIMESTAMP" > ? ORDER By "TIMESTAMP" ASC"
2016-07-18 20:00:15 DEBUG DBX2Proxy:491 - action=dbinput_tail_iterator_get_checkpoint_from_client checkpoint_value=1468861158000
2016-07-18 20:00:15 DEBUG Connector:291 - Database connection was established correctly: jdbcUrlFormat = [jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb)))], username = [TECH_ARCSIGHT].
2016-07-18 20:00:15 DEBUG Connector:241 - The database connection information: jdbcDriverClass = [oracle.jdbc.OracleDriver], jdbcUrlFormat = [jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb)))], username = [TECH_ARCSIGHT].
2016-07-18 20:00:15 DEBUG Connector:240 - SSL: false
2016-07-18 20:00:15 DEBUG Connector:183 - datasource=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))):jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))):TECH_ARCSIGHT:1651651705 driver=oracle.jdbc.OracleDriver
2016-07-18 20:00:15 DEBUG Connector:183 - datasource=oracledb_connection:jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))):TECH_ARCSIGHT:-1993644591 driver=oracle.jdbc.OracleDriver
2016-07-18 20:00:15 DEBUG Connector:180 - data_source_size=2
2016-07-18 20:00:15 DEBUG DefaultDBX2Input:342 - action=build_configuration_for_input params={iter_size=300, ui_query_table=NULL, ui_query_schema=NULL, ui_query_catalog=NULL, tail_follow_only=1, max_rows=50000, input_timestamp_column_name=TIMESTAMP, query=SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF'), mode=tail, ui_query_mode=advanced, output_timestamp_format=yyyy-MM-dd HH:mm:ss.SSS, connection=oracledb_connection, enable_query_wrapping=True, tail_rising_column_name=TIMESTAMP, tail_rising_column_checkpoint_value=1468861158000}
2016-07-18 20:00:15 DEBUG Connector:122 - DB pamameters while creating Connector: {TRANS_OBJECT_ID=7e231619-ba98-4f7f-b043-d74cccc0d34c, maxWaitMillis=120000, jdbcUrlSSLFormat=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))), maxIdleConn=8, serviceClass=com.splunk.dbx2.OracleJDBC, cwallet_location=/home/oracle/cwallet.sso, readonly=False, useConnectionPool=1, testQuery=SELECT 1 FROM DUAL, maxTotalConn=16, jdbcUrlFormat=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))), connection=oracledb_connection, jdbcDriverClass=oracle.jdbc.OracleDriver, maxConnLifetimeMillis=120000, username=TECH_ARCSIGHT}
2016-07-18 20:00:15 DEBUG ServiceResponder:416 - action=request_param index=3 type='class java.lang.String' value=mi_input://auditdb_events
2016-07-18 20:00:15 DEBUG ServiceResponder:416 - action=request_param index=2 type='class java.lang.String' value=admin
2016-07-18 20:00:15 DEBUG ServiceResponder:416 - action=request_param index=1 type='interface java.util.Map' value={iter_size=300, ui_query_table=NULL, ui_query_schema=NULL, ui_query_catalog=NULL, tail_follow_only=1, max_rows=50000, input_timestamp_column_name=TIMESTAMP, query=SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF'), mode=tail, ui_query_mode=advanced, output_timestamp_format=yyyy-MM-dd HH:mm:ss.SSS, connection=oracledb_connection, enable_query_wrapping=True, tail_rising_column_name=TIMESTAMP, tail_rising_column_checkpoint_value=1468861158000}
2016-07-18 20:00:15 DEBUG ServiceResponder:416 - action=request_param index=0 type='interface java.util.Map' value={TRANS_OBJECT_ID=7e231619-ba98-4f7f-b043-d74cccc0d34c, maxWaitMillis=120000, jdbcUrlSSLFormat=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))), maxIdleConn=8, serviceClass=com.splunk.dbx2.OracleJDBC, cwallet_location=/home/oracle/cwallet.sso, readonly=False, useConnectionPool=1, testQuery=SELECT 1 FROM DUAL, maxTotalConn=16, jdbcUrlFormat=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))), connection=oracledb_connection, jdbcDriverClass=oracle.jdbc.OracleDriver, maxConnLifetimeMillis=120000, username=TECH_ARCSIGHT}
2016-07-18 20:00:15 DEBUG ServiceResponder:406 - action=process_request request=dbinputTailIterator
2016-07-18 20:00:15 DEBUG ServiceSocket:76 - ServiceSocket session [1093650312] has received request message [dbinputTailIterator].
2016-07-18 20:00:15 DEBUG ServiceSocket:53 - New websocket session [1093650312] is open.
2016-07-18 20:00:15 DEBUG ServiceResponder:416 - action=request_param index=0 type='class [Ljava.lang.String;' value=[HOSTNAME, HOME, SPLUNK_WEB_NAME, SPLUNK_HOME, SPLUNK_SERVER_NAME, TERM_SESSION_ID]
2016-07-18 20:00:15 DEBUG ServiceResponder:406 - action=process_request request=getServerStatus
2016-07-18T20:00:15+0300 [INFO] [health_logger.py], line 193: CONNECTION=oracledb_connection DATABASE=oracledb DATABASE_TYPE=oracle DB_JDBC_URL=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))) DB_SERVER=oracle-db.my.domain DB_USER=TECH_ARCSIGHT FUNCTION=py_dbinput JDBC_USE_SSL=False LABEL=dbinput LOGIN_USER=admin STATE=started TRANS_OBJECT_ID=7e231619-ba98-4f7f-b043-d74cccc0d34c
2016-07-18 20:00:15 INFO  HealthLogger:193 - FREE_MEMORY=284760256 FUNCTION=getServerStatus LABEL=JP MAX_MEMORY=3162177536 PROTOCOL=HTTP STATE=started TOTAL_MEMORY=329232384 UPTIME=33464771 task_id=ec8c5ddd-2def-449f-bfec-67916aa05ea9
2016-07-18 20:00:15 DEBUG ServiceResponder:416 - action=request_param index=0 type='class [Ljava.lang.String;' value=[SPLUNK_HOME]
2016-07-18 20:00:15 DEBUG ServiceResponder:406 - action=process_request request=getServerStatus
2016-07-18T20:00:14+0300 [DEBUG] [mi_base.py], line 127: action=test_rpc_server_status
2016-07-18T20:00:14+0300 [DEBUG] [mi_base.py], line 106: action=skip_clustering_check cause=search_head_clustering_not_enabled_for_modular_input input_name=mi_input://auditdb_events
2016-07-18T20:00:14+0300 [DEBUG] [shc_cluster_config.py], line 16 : action=retrieve_shc_clustering clustering_mode=disabled clustering_enabled=False
2016-07-18T20:00:14+0300 [DEBUG] [shc_cluster_config.py], line 23 : action=test_if_enterprise_product product_type=enterprise result=True
2016-07-18T20:00:14+0300 [DEBUG] [dbxcrypto.py], line 37 : action=decrypt_data command="openssl aes-256-cbc -d -base64 -pass file:/opt/splunk/etc/apps/splunk_app_db_connect/certs/identity.dat"
2016-07-18T20:00:14+0300 [DEBUG] [connection_info.py], line 14 : action=get_connection_info name=oracledb_connection
2016-07-18 19:59:28 DEBUG ServiceSocket:199 - Websocket session [1093671466] is closed as status [1000], reason [null].
2016-07-18T19:59:28+0300 [INFO] [health_logger.py], line 193: CONNECTION=oracledb_connection DATABASE=oracledb DATABASE_TYPE=oracle DB_JDBC_URL=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))) DB_SERVER=oracle-db.my.domain DB_USER=TECH_ARCSIGHT DURATION=12.6247599125 ERROR_CODE=000 FUNCTION=py_dbinput JDBC_USE_SSL=False LABEL=dbinput LOGIN_USER=admin MEMORY_USED=1136000 STATE=completed TRANS_OBJECT_ID=bda1a71f-c4b2-4db3-87dd-10738877ba7f
2016-07-18T19:59:28+0300 [DEBUG] [mi_input.py], line 163: action=dbinput_time execution_duration="14.017486 seconds" dbinput="mi_input://auditdb_events" max_query_timeout="3600 seconds"
2016-07-18T19:59:28+0300 [INFO] [mi_input.py], line 153: action=complete_dbinput dbinput="mi_input://auditdb_events"
2016-07-18T19:59:28+0300 [DEBUG] [mi_input.py], line 54 : action=finish_running_tail_input dbinput="mi_input://auditdb_events"
2016-07-18T19:59:28+0300 [DEBUG] [websocket.py], line 542: length_of_data_sent=8
2016-07-18T19:59:28+0300 [INFO] [mi_input.py], line 81 : action=rising_column_checkpoint_updated dbinput="mi_input://auditdb_events" checkpoint=1468861158000
2016-07-18 19:59:27 DEBUG ServiceSocket:176 - Cancelling Timer task for session: 1093671466
2016-07-18 19:59:27 INFO  TailInputResultSetIterator:68 - action=tail_input_return_checkpoint_value checkpoint_value="1468861158000"
2016-07-18 19:59:27 DEBUG ResultSetIterator:153 - action=result_set_metadata column_count=29
2016-07-18 19:59:27 INFO  TailInputResultSetIterator:68 - action=tail_input_return_checkpoint_value checkpoint_value="1468861144000"
2016-07-18 19:59:27 DEBUG ResultSetIterator:153 - action=result_set_metadata column_count=29
2016-07-18 19:59:27 DEBUG DefaultDBX2Input:30 -     duration=11 splunk-user=admin   connection=oracledb_connection  rows=50000  params=[1468861098000]  sql=SELECT * FROM (SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF')) t WHERE "TIMESTAMP" > ? ORDER By "TIMESTAMP" ASC
2016-07-18T19:59:27+0300 [DEBUG] [mi_input.py], line 76 : action=updating_rising_column_checkpoint dbinput="mi_input://auditdb_events" checkpoint=1468861158000
2016-07-18T19:59:27+0300 [DEBUG] [modular_input_event_writer.py], line 63 : action=finish_printing_csv_to_event_stream dbinput="mi_input://auditdb_events" input_mode=tail
2016-07-18T19:59:27+0300 [DEBUG] [modular_input_event_writer.py], line 12 : </stream>
2016-07-18T19:59:27+0300 [DEBUG] [modular_input_event_writer.py], line 41 : <event stanza="mi_input://auditdb_events"><time>"2016-07-18 19:59:18.000"</time>
<data>"2016-07-18 19:59:18.000" TIMESTAMP="2016-07-18 19:59:18.0", OS_USERNAME="test1", USERNAME="TEST1", USERHOST="msk-zabbixpr02.my.domain", TERMINAL="NULL", OWNER="NULL", OBJ_NAME="NULL", ACTION="100", ACTION_NAME="LOGON", NEW_OWNER="NULL", NEW_NAME="NULL", OBJ_PRIVILEGE="NULL", SYS_PRIVILEGE="NULL", ADMIN_OPTION="NULL", SES_ACTIONS="NULL", LOGOFF_TIME="NULL", LOGOFF_LREAD="NULL", LOGOFF_PREAD="NULL", LOGOFF_LWRITE="NULL", LOGOFF_DLOCK="NULL", COMMENT_TEXT="Authenticated by: DATABASE; Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.18.2.100)(PORT=40905))", SESSIONID="371849983", ENTRYID="1", STATEMENTID="1", RETURNCODE="0", PRIV_USED="CREATE SESSION", CLIENT_ID="NULL", ECONTEXT_ID="NULL", SESSION_CPU="NULL"</data></event>
2016-07-18T19:59:27+0300 [DEBUG] [modular_input_event_writer.py], line 41 : <event stanza="mi_input://auditdb_events"><time>"2016-07-18 19:59:18.000"</time>
<data>"2016-07-18 19:59:18.000" TIMESTAMP="2016-07-18 19:59:18.0", OS_USERNAME="TEST$", USERNAME="TEST", USERHOST="TEST", TERMINAL="unknown", OWNER="NULL", OBJ_NAME="NULL", ACTION="100", ACTION_NAME="LOGON", NEW_OWNER="NULL", NEW_NAME="NULL", OBJ_PRIVILEGE="NULL", SYS_PRIVILEGE="NULL", ADMIN_OPTION="NULL", SES_ACTIONS="NULL", LOGOFF_TIME="NULL", LOGOFF_LREAD="NULL", LOGOFF_PREAD="NULL", LOGOFF_LWRITE="NULL", LOGOFF_DLOCK="NULL", COMMENT_TEXT="Authenticated by: DATABASE; Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.18.23.13)(PORT=57721))", SESSIONID="371849956", ENTRYID="1", STATEMENTID="1", RETURNCODE="28001", PRIV_USED="NULL", CLIENT_ID="NULL", ECONTEXT_ID="NULL", SESSION_CPU="NULL"</data></event>
...

inputs.conf

[mi_input://auditdb_events]
connection = oracledb_connection
index = myindex
input_timestamp_column_name = TIMESTAMP
interval = 60
max_rows = 50000
mode = tail
output_timestamp_format = yyyy-MM-dd HH:mm:ss.SSS
query = SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF')
source = oracle:cft-stb:sys:dba_audit_trail
sourcetype = dbx2
tail_follow_only = 1
tail_rising_column_checkpoint_value = 1468861158000
tail_rising_column_name = TIMESTAMP
ui_query_catalog = NULL
ui_query_mode = advanced
ui_query_schema = NULL
ui_query_table = NULL
disabled = 0

If i disable and enable this input at splunk web DB Connect starts to retrieve events again.

[bspargur1]

Based on your memory usage I am thinking you are pulling back a decent amount of data. Your checkpoint value is quite high suggesting your data set is very large and this is not a new forwarder, so this likely worked before. I am guessing the data set has gone up recently or there was an issue that caused you to be able to pull for a period of time, which caused this to come to a head. Try upgrading to 2.4. The below bug, which was fixed may be what is... well... bugging you. 🙂

DBX-2759    dbinput tail won't index a large set of row data, avg 150K/event, total 3000+ event. 

[jcoates_splunk]

I don't see an error, so I assume this isn't auto-disabling but rather is confused about what rows you want to collect. I would set tail_follow_only to 0. This option was intended to emulate the behavior of Splunk UFs with syslog files, but it's just confusing in database tailing context and has been removed from the UI in 2.3.0.

[bspargur1]

2016-07-18T19:59:28+0300 [INFO] [health_logger.py], line 193: CONNECTION=oracledb_connection DATABASE=oracledb DATABASE_TYPE=oracle DB_JDBC_URL=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))) DB_SERVER=oracle-db.my.domain DB_USER=TECH_ARCSIGHT DURATION=12.6247599125 ERROR_CODE=000 FUNCTION=py_dbinput JDBC_USE_SSL=False LABEL=dbinput LOGIN_USER=admin MEMORY_USED=1136000 STATE=completed TRANS_OBJECT_ID=bda1a71f-c4b2-4db3-87dd-10738877ba7f

It is likely timing out due to him reaching the timeout on the DB server or because the next iteration runs terminating the previous session. That would be my guess.

[bspargur1]

ERROR_CODE=000 is the error in the log above