Scheduled dbxquery fires sql twice to a Oracle DB

[stanwin]

Hello

Using DBX 3.1.3 on Splunk 7.1.3

We have a issue where dbx fires the scheduled SQL query twice to the database instead of just once.

This is confirmed by the oracle DB team with OEM & controlled tests across environments for this area.

The requests are sent milliseconds apart.

The ingest is setup as a Scheduled search on search heads using collect command.

p.s. There are several other Oracle DBs which are being tail'ed the same way & we can see only one dbxquery being fired to them. they dont have this issue.

Disabling & enabling the saved search and /or the connection hasn't helped.

Can see that the requests are logged once by splunk scheduler ( for the Scheduled search) but logged twice in dbx logs( sbx executing the query to DB).

The queries are also fired from the same SH just milliseconds apart as mentioned.

[stanwin · Accepted solution]

This issue was resolved after changing execution phased_execution_mode to singlethreaded

---

https://docs.splunk.com/Documentation/Splunk/7.1.3/ReleaseNotes/Knownissues#Search_issues
SPL-155648

New phased_execution_mode is spawning extra processes for custom search commands

Workaround:
If the custom search needs to run only once, disable the multithread for all searches.

$SPLUNK/etc/system/local stopped the issue from occurring. [search] phased_execution_mode = singlethreaded

Apply this workaround especially for deployment using ITSI, as the bug causes double backfill of the ITSI Episodes.

Can you change this on your SHC's limits.conf (via deployer):

https://docs.splunk.com/Documentation/Splunk/7.1.3/Admin/Limitsconf
[search]
phased_execution_mode = singlethreaded

[stanwin]

update:

Splunk support was able to duplicate the issue at their test bed & also stop the behaviour by piping below command at the end.

Splunk search optimizer apparently tries to optimize the query & for unknown reason fires it twice.

| noop search_optimization=f

This workaround does not work for me though.

Splunk support has forwarded the case to Splunk engineering.

[stanwin]

opened a case for this. Will update with any findings.