Please help me to understand why my db input stops to retrieve events from Oracle dba_audit_trail system view.
Here is a piece of Splunk DB Connect 2 logs with last execution of the db input trail iterator ( the search command is
index=_internal (source=*dbx2.log OR source=*rpc.log OR source=*health.log )
and so on...
2016-07-18 20:01:45 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18T20:01:35+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:01:35 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18T20:01:25+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:01:25 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18T20:01:15+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:01:15 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18T20:01:05+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:01:05 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18T20:00:55+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:00:55 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18T20:00:45+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:00:45 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18T20:00:35+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:00:35 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18 20:00:25 DEBUG ServiceSocket:260 - Timer task running for session: 1093650312
2016-07-18 20:00:25 DEBUG ResultSetIterator:153 - action=result_set_metadata column_count=29
2016-07-18T20:00:25+0300 [DEBUG] [ws.py], line 396: Input Service Received RPC ping
2016-07-18 20:00:25 INFO HealthLogger:193 - DB_JDBC_URL=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))) FREE_MEMORY=283981504 FUNCTION=dbinputTailIterator LABEL=JP MAX_MEMORY=3162177536 PROTOCOL=HTTP SQL="SELECT * FROM (SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF')) t WHERE \"TIMESTAMP\" > ? ORDER By \"TIMESTAMP\" ASC" STATE=completed TOTAL_MEMORY=329232384 TRANS_OBJECT_ID=7e231619-ba98-4f7f-b043-d74cccc0d34c UPTIME=33474978 task_id=7f15a1e3-11c3-4124-8fbc-889a583d684a
2016-07-18 20:00:25 DEBUG DefaultDBX2Input:30 - duration=9 splunk-user=admin connection=oracledb_connection rows=50000 params=[1468861158000] sql=SELECT * FROM (SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF')) t WHERE "TIMESTAMP" > ? ORDER By "TIMESTAMP" ASC
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 542: length_of_data_sent=1593
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 523: -----------------------
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 511: Upgrade: WebSocket
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 511: Sec-WebSocket-Accept: OqtiOHz9PTRolwxclcsPwZSKT10=
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 511: Connection: Upgrade
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 511: Date: Mon, 18 Jul 2016 17:00:15 GMT
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 511: HTTP/1.1 101 Switching Protocols
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 504: --- response header ---
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 469: -----------------------
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 468: GET /ws HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: localhost:9999
Origin: http://localhost:9999
Sec-WebSocket-Key: pjyTZmA6TXyX4GAQ9N3NWw==
Sec-WebSocket-Version: 13
Collapse
2016-07-18T20:00:15+0300 [DEBUG] [websocket.py], line 467: --- request header ---
2016-07-18T20:00:15+0300 [DEBUG] [mi_input.py], line 52 : action=run_tail_input dbinput="mi_input://auditdb_events"
2016-07-18T20:00:15+0300 [DEBUG] [mi_input.py], line 94 : action=get_rising_column_checkpoint_for_input dbinput="mi_input://auditdb_events" checkpoint=1468861158000
2016-07-18T20:00:15+0300 [DEBUG] [mi_input.py], line 106: action=get_input_websocket websocket_url=ws_service://localhost:9999/ws
2016-07-18T20:00:15+0300 [DEBUG] [mi_input.py], line 117: action=get_enable_query_wrapping_for_connection connection=oracledb_connection enable_query_wrapping=True
2016-07-18T20:00:15+0300 [INFO] [mi_input.py], line 136: action=start_executing_dbinput dbinput="mi_input://auditdb_events"
2016-07-18T20:00:15+0300 [DEBUG] [health_logger.py], line 132: health logger is ON
2016-07-18T20:00:15+0300 [DEBUG] [health_logger.py], line 108: action=check_health_logger_config user=admin, role=admin
2016-07-18 20:00:15 INFO HealthLogger:193 - DB_JDBC_URL=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))) FREE_MEMORY=284249248 FUNCTION=dbinputTailIterator LABEL=JP MAX_MEMORY=3162177536 PROTOCOL=HTTP SQL="SELECT * FROM (SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF')) t WHERE \"TIMESTAMP\" > ? ORDER By \"TIMESTAMP\" ASC" STATE=started TOTAL_MEMORY=329232384 TRANS_OBJECT_ID=7e231619-ba98-4f7f-b043-d74cccc0d34c UPTIME=33465008 task_id=7f15a1e3-11c3-4124-8fbc-889a583d684a
2016-07-18 20:00:15 INFO HealthLogger:193 - FREE_MEMORY=284519280 FUNCTION=getServerStatus LABEL=JP MAX_MEMORY=3162177536 PROTOCOL=HTTP STATE=completed TOTAL_MEMORY=329232384 UPTIME=33464802 task_id=ce79b1f1-1a3e-4271-a71b-e81f3bdb13e7
2016-07-18 20:00:15 INFO HealthLogger:193 - FREE_MEMORY=284519280 FUNCTION=getServerStatus LABEL=JP MAX_MEMORY=3162177536 PROTOCOL=HTTP STATE=started TOTAL_MEMORY=329232384 UPTIME=33464801 task_id=ce79b1f1-1a3e-4271-a71b-e81f3bdb13e7
2016-07-18 20:00:15 INFO HealthLogger:193 - FREE_MEMORY=284760256 FUNCTION=getServerStatus LABEL=JP MAX_MEMORY=3162177536 PROTOCOL=HTTP STATE=completed TOTAL_MEMORY=329232384 UPTIME=33464775 task_id=ec8c5ddd-2def-449f-bfec-67916aa05ea9
2016-07-18 20:00:15 DEBUG Connector:183 - datasource=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))):jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))):TECH_ARCSIGHT:1651651705 driver=oracle.jdbc.OracleDriver
2016-07-18 20:00:15 DEBUG Connector:183 - datasource=oracledb_connection:jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))):TECH_ARCSIGHT:-1993644591 driver=oracle.jdbc.OracleDriver
2016-07-18 20:00:15 DEBUG Connector:180 - data_source_size=2
2016-07-18 20:00:15 DEBUG DefaultDBX2Input:316 - action=run_tail_input_iterator query="SELECT * FROM (SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF')) t WHERE "TIMESTAMP" > ? ORDER By "TIMESTAMP" ASC" checkpoint=1468861158000 rising_column=TIMESTAMP rising_column_no=1
2016-07-18 20:00:15 DEBUG DBX2Proxy:495 - action=dbinput_tail_iterator_get_query_from_client query="SELECT * FROM (SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF')) t WHERE "TIMESTAMP" > ? ORDER By "TIMESTAMP" ASC"
2016-07-18 20:00:15 DEBUG DBX2Proxy:491 - action=dbinput_tail_iterator_get_checkpoint_from_client checkpoint_value=1468861158000
2016-07-18 20:00:15 DEBUG Connector:291 - Database connection was established correctly: jdbcUrlFormat = [jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb)))], username = [TECH_ARCSIGHT].
2016-07-18 20:00:15 DEBUG Connector:241 - The database connection information: jdbcDriverClass = [oracle.jdbc.OracleDriver], jdbcUrlFormat = [jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb)))], username = [TECH_ARCSIGHT].
2016-07-18 20:00:15 DEBUG Connector:240 - SSL: false
2016-07-18 20:00:15 DEBUG Connector:183 - datasource=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))):jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))):TECH_ARCSIGHT:1651651705 driver=oracle.jdbc.OracleDriver
2016-07-18 20:00:15 DEBUG Connector:183 - datasource=oracledb_connection:jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))):TECH_ARCSIGHT:-1993644591 driver=oracle.jdbc.OracleDriver
2016-07-18 20:00:15 DEBUG Connector:180 - data_source_size=2
2016-07-18 20:00:15 DEBUG DefaultDBX2Input:342 - action=build_configuration_for_input params={iter_size=300, ui_query_table=NULL, ui_query_schema=NULL, ui_query_catalog=NULL, tail_follow_only=1, max_rows=50000, input_timestamp_column_name=TIMESTAMP, query=SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF'), mode=tail, ui_query_mode=advanced, output_timestamp_format=yyyy-MM-dd HH:mm:ss.SSS, connection=oracledb_connection, enable_query_wrapping=True, tail_rising_column_name=TIMESTAMP, tail_rising_column_checkpoint_value=1468861158000}
2016-07-18 20:00:15 DEBUG Connector:122 - DB pamameters while creating Connector: {TRANS_OBJECT_ID=7e231619-ba98-4f7f-b043-d74cccc0d34c, maxWaitMillis=120000, jdbcUrlSSLFormat=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))), maxIdleConn=8, serviceClass=com.splunk.dbx2.OracleJDBC, cwallet_location=/home/oracle/cwallet.sso, readonly=False, useConnectionPool=1, testQuery=SELECT 1 FROM DUAL, maxTotalConn=16, jdbcUrlFormat=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))), connection=oracledb_connection, jdbcDriverClass=oracle.jdbc.OracleDriver, maxConnLifetimeMillis=120000, username=TECH_ARCSIGHT}
2016-07-18 20:00:15 DEBUG ServiceResponder:416 - action=request_param index=3 type='class java.lang.String' value=mi_input://auditdb_events
2016-07-18 20:00:15 DEBUG ServiceResponder:416 - action=request_param index=2 type='class java.lang.String' value=admin
2016-07-18 20:00:15 DEBUG ServiceResponder:416 - action=request_param index=1 type='interface java.util.Map' value={iter_size=300, ui_query_table=NULL, ui_query_schema=NULL, ui_query_catalog=NULL, tail_follow_only=1, max_rows=50000, input_timestamp_column_name=TIMESTAMP, query=SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF'), mode=tail, ui_query_mode=advanced, output_timestamp_format=yyyy-MM-dd HH:mm:ss.SSS, connection=oracledb_connection, enable_query_wrapping=True, tail_rising_column_name=TIMESTAMP, tail_rising_column_checkpoint_value=1468861158000}
2016-07-18 20:00:15 DEBUG ServiceResponder:416 - action=request_param index=0 type='interface java.util.Map' value={TRANS_OBJECT_ID=7e231619-ba98-4f7f-b043-d74cccc0d34c, maxWaitMillis=120000, jdbcUrlSSLFormat=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))), maxIdleConn=8, serviceClass=com.splunk.dbx2.OracleJDBC, cwallet_location=/home/oracle/cwallet.sso, readonly=False, useConnectionPool=1, testQuery=SELECT 1 FROM DUAL, maxTotalConn=16, jdbcUrlFormat=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))), connection=oracledb_connection, jdbcDriverClass=oracle.jdbc.OracleDriver, maxConnLifetimeMillis=120000, username=TECH_ARCSIGHT}
2016-07-18 20:00:15 DEBUG ServiceResponder:406 - action=process_request request=dbinputTailIterator
2016-07-18 20:00:15 DEBUG ServiceSocket:76 - ServiceSocket session [1093650312] has received request message [dbinputTailIterator].
2016-07-18 20:00:15 DEBUG ServiceSocket:53 - New websocket session [1093650312] is open.
2016-07-18 20:00:15 DEBUG ServiceResponder:416 - action=request_param index=0 type='class [Ljava.lang.String;' value=[HOSTNAME, HOME, SPLUNK_WEB_NAME, SPLUNK_HOME, SPLUNK_SERVER_NAME, TERM_SESSION_ID]
2016-07-18 20:00:15 DEBUG ServiceResponder:406 - action=process_request request=getServerStatus
2016-07-18T20:00:15+0300 [INFO] [health_logger.py], line 193: CONNECTION=oracledb_connection DATABASE=oracledb DATABASE_TYPE=oracle DB_JDBC_URL=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))) DB_SERVER=oracle-db.my.domain DB_USER=TECH_ARCSIGHT FUNCTION=py_dbinput JDBC_USE_SSL=False LABEL=dbinput LOGIN_USER=admin STATE=started TRANS_OBJECT_ID=7e231619-ba98-4f7f-b043-d74cccc0d34c
2016-07-18 20:00:15 INFO HealthLogger:193 - FREE_MEMORY=284760256 FUNCTION=getServerStatus LABEL=JP MAX_MEMORY=3162177536 PROTOCOL=HTTP STATE=started TOTAL_MEMORY=329232384 UPTIME=33464771 task_id=ec8c5ddd-2def-449f-bfec-67916aa05ea9
2016-07-18 20:00:15 DEBUG ServiceResponder:416 - action=request_param index=0 type='class [Ljava.lang.String;' value=[SPLUNK_HOME]
2016-07-18 20:00:15 DEBUG ServiceResponder:406 - action=process_request request=getServerStatus
2016-07-18T20:00:14+0300 [DEBUG] [mi_base.py], line 127: action=test_rpc_server_status
2016-07-18T20:00:14+0300 [DEBUG] [mi_base.py], line 106: action=skip_clustering_check cause=search_head_clustering_not_enabled_for_modular_input input_name=mi_input://auditdb_events
2016-07-18T20:00:14+0300 [DEBUG] [shc_cluster_config.py], line 16 : action=retrieve_shc_clustering clustering_mode=disabled clustering_enabled=False
2016-07-18T20:00:14+0300 [DEBUG] [shc_cluster_config.py], line 23 : action=test_if_enterprise_product product_type=enterprise result=True
2016-07-18T20:00:14+0300 [DEBUG] [dbxcrypto.py], line 37 : action=decrypt_data command="openssl aes-256-cbc -d -base64 -pass file:/opt/splunk/etc/apps/splunk_app_db_connect/certs/identity.dat"
2016-07-18T20:00:14+0300 [DEBUG] [connection_info.py], line 14 : action=get_connection_info name=oracledb_connection
2016-07-18 19:59:28 DEBUG ServiceSocket:199 - Websocket session [1093671466] is closed as status [1000], reason [null].
2016-07-18T19:59:28+0300 [INFO] [health_logger.py], line 193: CONNECTION=oracledb_connection DATABASE=oracledb DATABASE_TYPE=oracle DB_JDBC_URL=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-db.my.domain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oracledb))) DB_SERVER=oracle-db.my.domain DB_USER=TECH_ARCSIGHT DURATION=12.6247599125 ERROR_CODE=000 FUNCTION=py_dbinput JDBC_USE_SSL=False LABEL=dbinput LOGIN_USER=admin MEMORY_USED=1136000 STATE=completed TRANS_OBJECT_ID=bda1a71f-c4b2-4db3-87dd-10738877ba7f
2016-07-18T19:59:28+0300 [DEBUG] [mi_input.py], line 163: action=dbinput_time execution_duration="14.017486 seconds" dbinput="mi_input://auditdb_events" max_query_timeout="3600 seconds"
2016-07-18T19:59:28+0300 [INFO] [mi_input.py], line 153: action=complete_dbinput dbinput="mi_input://auditdb_events"
2016-07-18T19:59:28+0300 [DEBUG] [mi_input.py], line 54 : action=finish_running_tail_input dbinput="mi_input://auditdb_events"
2016-07-18T19:59:28+0300 [DEBUG] [websocket.py], line 542: length_of_data_sent=8
2016-07-18T19:59:28+0300 [INFO] [mi_input.py], line 81 : action=rising_column_checkpoint_updated dbinput="mi_input://auditdb_events" checkpoint=1468861158000
2016-07-18 19:59:27 DEBUG ServiceSocket:176 - Cancelling Timer task for session: 1093671466
2016-07-18 19:59:27 INFO TailInputResultSetIterator:68 - action=tail_input_return_checkpoint_value checkpoint_value="1468861158000"
2016-07-18 19:59:27 DEBUG ResultSetIterator:153 - action=result_set_metadata column_count=29
2016-07-18 19:59:27 INFO TailInputResultSetIterator:68 - action=tail_input_return_checkpoint_value checkpoint_value="1468861144000"
2016-07-18 19:59:27 DEBUG ResultSetIterator:153 - action=result_set_metadata column_count=29
2016-07-18 19:59:27 DEBUG DefaultDBX2Input:30 - duration=11 splunk-user=admin connection=oracledb_connection rows=50000 params=[1468861098000] sql=SELECT * FROM (SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF')) t WHERE "TIMESTAMP" > ? ORDER By "TIMESTAMP" ASC
2016-07-18T19:59:27+0300 [DEBUG] [mi_input.py], line 76 : action=updating_rising_column_checkpoint dbinput="mi_input://auditdb_events" checkpoint=1468861158000
2016-07-18T19:59:27+0300 [DEBUG] [modular_input_event_writer.py], line 63 : action=finish_printing_csv_to_event_stream dbinput="mi_input://auditdb_events" input_mode=tail
2016-07-18T19:59:27+0300 [DEBUG] [modular_input_event_writer.py], line 12 : </stream>
2016-07-18T19:59:27+0300 [DEBUG] [modular_input_event_writer.py], line 41 : <event stanza="mi_input://auditdb_events"><time>"2016-07-18 19:59:18.000"</time>
<data>"2016-07-18 19:59:18.000" TIMESTAMP="2016-07-18 19:59:18.0", OS_USERNAME="test1", USERNAME="TEST1", USERHOST="msk-zabbixpr02.my.domain", TERMINAL="NULL", OWNER="NULL", OBJ_NAME="NULL", ACTION="100", ACTION_NAME="LOGON", NEW_OWNER="NULL", NEW_NAME="NULL", OBJ_PRIVILEGE="NULL", SYS_PRIVILEGE="NULL", ADMIN_OPTION="NULL", SES_ACTIONS="NULL", LOGOFF_TIME="NULL", LOGOFF_LREAD="NULL", LOGOFF_PREAD="NULL", LOGOFF_LWRITE="NULL", LOGOFF_DLOCK="NULL", COMMENT_TEXT="Authenticated by: DATABASE; Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.18.2.100)(PORT=40905))", SESSIONID="371849983", ENTRYID="1", STATEMENTID="1", RETURNCODE="0", PRIV_USED="CREATE SESSION", CLIENT_ID="NULL", ECONTEXT_ID="NULL", SESSION_CPU="NULL"</data></event>
2016-07-18T19:59:27+0300 [DEBUG] [modular_input_event_writer.py], line 41 : <event stanza="mi_input://auditdb_events"><time>"2016-07-18 19:59:18.000"</time>
<data>"2016-07-18 19:59:18.000" TIMESTAMP="2016-07-18 19:59:18.0", OS_USERNAME="TEST$", USERNAME="TEST", USERHOST="TEST", TERMINAL="unknown", OWNER="NULL", OBJ_NAME="NULL", ACTION="100", ACTION_NAME="LOGON", NEW_OWNER="NULL", NEW_NAME="NULL", OBJ_PRIVILEGE="NULL", SYS_PRIVILEGE="NULL", ADMIN_OPTION="NULL", SES_ACTIONS="NULL", LOGOFF_TIME="NULL", LOGOFF_LREAD="NULL", LOGOFF_PREAD="NULL", LOGOFF_LWRITE="NULL", LOGOFF_DLOCK="NULL", COMMENT_TEXT="Authenticated by: DATABASE; Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.18.23.13)(PORT=57721))", SESSIONID="371849956", ENTRYID="1", STATEMENTID="1", RETURNCODE="28001", PRIV_USED="NULL", CLIENT_ID="NULL", ECONTEXT_ID="NULL", SESSION_CPU="NULL"</data></event>
...
inputs.conf
[mi_input://auditdb_events]
connection = oracledb_connection
index = myindex
input_timestamp_column_name = TIMESTAMP
interval = 60
max_rows = 50000
mode = tail
output_timestamp_format = yyyy-MM-dd HH:mm:ss.SSS
query = SELECT /*+ PARALLEL (8) */ TIMESTAMP,OS_USERNAME,USERNAME,USERHOST,TERMINAL,OWNER,OBJ_NAME,ACTION,ACTION_NAME,NEW_OWNER,NEW_NAME,OBJ_PRIVILEGE,SYS_PRIVILEGE,ADMIN_OPTION,SES_ACTIONS,LOGOFF_TIME,LOGOFF_LREAD,LOGOFF_PREAD,LOGOFF_LWRITE,LOGOFF_DLOCK,COMMENT_TEXT,SESSIONID,ENTRYID,STATEMENTID,RETURNCODE,PRIV_USED,CLIENT_ID,ECONTEXT_ID,SESSION_CPU FROM SYS.DBA_AUDIT_TRAIL WHERE TIMESTAMP > TO_TIMESTAMP ('01-07-2016 00:00:00.000000', 'DD-MM-YYYY HH24:MI:SS.FF')
source = oracle:cft-stb:sys:dba_audit_trail
sourcetype = dbx2
tail_follow_only = 1
tail_rising_column_checkpoint_value = 1468861158000
tail_rising_column_name = TIMESTAMP
ui_query_catalog = NULL
ui_query_mode = advanced
ui_query_schema = NULL
ui_query_table = NULL
disabled = 0
If i disable and enable this input at splunk web DB Connect starts to retrieve events again.
... View more