Under Splunk DB Connect, we have data inputs created which will periodically pull data from our SQLServer database and put it into our indexes. The SQL queries used have a rising column which acts as a checkpoint column for these inputs. Things were working fine until KVStore went down on the Splunk server. To bring it up, we followed these steps which worked. Stop the search head that has the stale KV store member. Run the command splunk clean kvstore --local. Restart the search head. Run the command splunk show kvstore-status to verify synchronization. But, after doing this, Splunk DB Connect stopped writing any new data to the indexes. The DB Connect logs are filled with error that look like this for all data inputs: loading checkpoint for input title=CDA_Eportal_JobUserMapping loading checkpoint by checkpointKey=63863acecd230000be007648 from KV Store for input title=CDA_Eportal_JobUserMapping error loading checkpoint for input title=CDA_Eportal_JobUserMapping java.lang.RuntimeException: Can't find checkpoint for DB Input CDA_Eportal_JobUserMapping I am unable to even edit the data inputs to manually add checkpoints as they fail while saving. Is there any way to fix all checkpoints or clear all of them so that data gets written to the indexes again? What should I do to fix this issue?  

I need to backfill some missing data into the summary index. However, there are already a few data present in the same index. Therefore, I only want to backfill the remaining events, and the data that is already present should not be injected again. I am currently using the 'fill_summary_index.py' script, but during testing, it seems to inject duplicate data, indicating that the deduplication function is not working correctly in this script. Please help me by providing a proper script to address this issue.

Ever since upgrading to version 2.1.4 of Cofense Triage Add-On, we get hundreds of these errors in our _internal logs: 01-30-2024 13:20:42.094 -0500 ERROR ScriptRunner [1503161 TcpChannelThread] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py execute': /opt/splunk/etc/apps/TA-cofense-triage-add-on-for-splunk/bin/ta_cofense_triage_add_on_for_splunk/aob_py3/solnlib/utils.py:153: UserWarning: _get_all_passwords is deprecated, please use get_all_passwords_in_realm instead.   It does not affect log ingestion, but would like help in figuring out how to suppress the errors.

I am working on creating an alert from Splunk.  In my search I am creating a variable using eval, but that is not used in the result table.  But I would like to use it in the email subject and body.     index=applications sourcetype=aws:cloudwatchlogs ((Job="*prod-job1*") OR (Job="*prod-job2*")) | eval emailTime=strftime(now(),"%m/%d/%Y") | stats latest(_time) as latest(s3Partition) as s3Partition latest(field1) as field1 latest(field2) as field2 latest(emailTime) as emailTime by table_name | search field2 ="*" emailTime=* | eval diff=(field2-field1) | eval evt_time=strftime(_time, "%Y-%m-%d") | eval partition_date=substr(s3Partition, len("event_creation_time=")+1, len("yyyy-mm-dd")) | where isnotnull(table_name) and isnotnull(emailTime) and ( evt_time == partition_date ) | table table_name, field1, field2, diff | sort raw_table_name | rename table_name AS "Table Name" field1 AS "Field1 count" field2 AS "Field2 count" diff as "Count Difference"       I tried using it like  -    $result.partition_date$  and  $result.emailTime$    -    in the subject and body, but the value is not getting substituted -  it appears  empty in both the places. Is it possible to use this value in email without using it in the table for the alert? Thank you  

Our current SOAR servers, fresh install on AWS EC2s, 500's each night. Upon investigation, it looks like there's this error in the logs: File "/opt/soar/usr/python39/lib/python3.9/site-packages/psycopg2/__init__.py", line 127, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) django.db.utils.OperationalError: connection to server on socket "/tmp//.s.PGSQL.6432" failed: No such file or directory Is the server running locally and accepting connections on that socket? On a healthy server, that file is present. On a 500-error server, it's missing. Is there an explanation of why that might be going missing? Issue is temporarily resolved by stopping and starting phantom again.  I think it might be related to PostgreSQL or pgbouncer. 

I have a splunk distributed system with 3 indexers, 3 search heads, a manger, and 2 heavy forwarders.  I am attempting to deploy the DB Connect application to the HF and the SHC.  The SHC has 3 member nodes and the deployer on the manger node.  Ideally, this would all be done with ansible, sadly, the deployer gets in the way.  I can deploy to the HF with ansible, but the deployer keeps removing the db connect app on the SHC. That said, to deploy I install the app on the manager node, install the drivers, and then copy it to the ...shcluster/apps directory  and run the splunk schcluster apply command.   I've done this both manually and using ansible.   When I run the apply the deployer does not put the entire app on the search heads, it only puts the default and metadata directories on the Search heads in the splunk_app_db_connect directory.    When I go into the manage  apps on the GUI I see the app installed but it is not visible.  I would prefer not to use the GUI for management and perform all management task via the cli and ansible.  The code is stored in a version control system and gives not only control over the deployments but also trakcs who did what, when, why, and how.   So I guess there are multiple questions.   Why is the deployer not pushing the entire application to the search heads? How can I disable the deployer and just use ansible?