Splunk DB Connect: Error with details is empty, Slow index new data, Enscrypt/hass field before indexing

[dailv1808]

Hello Splunkers.

First of all, I'm sorry because my english is not good.

I am using Splunk DB Connect 2.4.1 on Splunk 7.2.6 and I got 3 problems need to help.

1. Error with detail is empty
Looking at _internal, I saw these following errors (error ="" )

2020-12-09T10:44:30+0700 [CRITICAL] [mi_input.py], line 61 : action=loading_input_data_failed input_mode=tail dbinput="mi_input://DATA" error=""
2020-12-09T10:44:30+0700 [CRITICAL] [ws.py], line 327: [DBInput Service] Exception encountered for entity-name = mi_input://DATA and type = input with error = .

2020-12-09T10:44:30+0700 [INFO] [mi_base.py], line 190: action=caught_exception_in_modular_input_with_retries modular_input=mi_input://DATA retrying="1 of 6" error=
Traceback (most recent call last):
  File "/u01/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/mi_base.py", line 183, in run
    checkpoint_value=checkpoint_value)
  File "/u01/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/health_logger.py", line 283, in wrapper
    return get_mdc(MDC_LOGGER).do_log(func, *args, **kwargs)
  File "/u01/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/health_logger.py", line 160, in do_log
    return func(*args, **kwargs)
  File "/u01/splunk/etc/apps/splunk_app_db_connect/bin/mi_input.py", line 205, in run
    _do_tail_mode(input_name, inputws, self.db, params, self.user_name, splunk_service, output_stream)
  File "/u01/splunk/etc/apps/splunk_app_db_connect/bin/mi_input.py", line 57, in _do_tail_mode
    inputws.doTail(db, params, user, stanza, callback=callback)
  File "/u01/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/ws.py", line 281, in doTail
    self.doInput("dbinputTailIterator", database, params, user, entityName, callback)
  File "/u01/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/ws.py", line 275, in doInput
    self.ws.run_forever(timeout=self.timeout)
  File "/u01/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/websocket.py", line 841, in run_forever
    self._callback(self.on_error, e)
  File "/u01/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/websocket.py", line 852, in _callback
    callback(self, *args)
  File "/u01/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/ws.py", line 328, in on_error
    raise Exception ("%s" % error)
Exception
Collapse

Have anyone seen this before? why it happened and how to fix them.

2. Database input job works very slow.

For some time, everything was OK, but all of sudden Splunk stopped indexing new data.

Looking at _internal, I saw only this one message:

2020-12-09T10:44:33+0700 [INFO] [mi_input.py], line 193: action=start_executing_dbinput dbinput="mi_input://DATA"

In case job works fine, it will have many massages like below:

2020-12-09T10:44:33+0700 [INFO] [mi_input.py], line 193: action=start_executing_dbinput dbinput="mi_input://DATA"

2020-12-09T10:44:29+0700 [INFO] [modular_input_event_writer.py], line 113: action=print_csv_from_jdbc_to_event_stream dbinput="mi_input://DATA" input_mode=tail events=300
2020-12-09T10:44:29+0700 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://DATA" checkpoint=8068170343

2020-12-09T10:45:52+0700 [INFO] [mi_input.py], line 193: action=complete_dbinput dbinput="mi_input://DATA"

I tried to check by query on database connect app interface, result very fast. So i think, database input job got problem.

3. Enscrypt/hass field before indexing

I am using Splunk DB Connect 2.4.1 on Splunk 7.2.6.

Some fields of data are case sentitive, ex: card_number. So i edited code in modular_input_event_writer.py  file in the DB connect app, it will hass card_number field to new field called hass_number. It work fine.

With Splunk DB Connect 3.x version. I cant enscrypt data field by the that way because the DB connect 3.x use java and python 3. It very different than 2.4.1 version. So is there any way to encrypt a data field before splunk indexing?

Thanks in advance.

[dailv1808 · Accepted solution]

Hi,

I resolved enscrypt/hass field before indexing by this way:

INGEST_EVAL = CARD_HASH=upper(sha256(replace(_raw, "^(.*)credit_card=\"(.*)\"$", "\2"))), _raw=replace(_raw, "^(.*credit_card)=(.*)$", "\1=".CARD_HASH)

[scelikok]

@dailv1808 , I would upgrade DBConnect3 app to the latest supported version. DBConnect 3 works much faster and stable than DBConnect2. This may resolve errors and slowness. 

Regarding card numbers you can use SEDCMD on props.conf to anonymize data. Of course you are limited by Regex SED capabilities. You can find details on below doc page.

https://docs.splunk.com/Documentation/Splunk/8.1.0/Data/Anonymizedata 

[dailv1808]

Thanks your response @scelikok 

I very want to update to DBConnect 3 but I need to handle the card number encryption first.

Using SEDCMD on props.conf look like masking function, not hashing. I need hashing instead of masking at index time.

Using a hash function (like MD5 or SHA) the values instead would fully protect the original value from being discovered, and still allows for accurate grouping and/or transaction operations on the sensitive field.