topic New application of the search head cannot make search data in Getting Data In

New application of the search head cannot make search data

chustar — Thu, 06 Jul 2017 02:46:17 GMT

I've created two splunk applications that work fine and can access the data on my indexers.

However, when I create a third application named "new_application" (from scratch or cloning an existing app) it cannot access any of the data.

While I can successfully load pregenerated data (using loadjob) it fails when I try to make a direct search.

I looked in the search log for all the indexers that were used and found this error:

dispatchRunner - RunDispatch::runDispatchThread threw error: Application does not exist: new_application

Is there an additional deployment or new config setting that needs to be changed when creating new applications?

edit:
This was due to the app not being replicated. I found the issue to be related to be related to the syncing the replication bundle:

DistributedBundleReplicationManager - Synchronous bundle replication to 4 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed_ms=10859, tar_elapsed_ms=551, bundle_file_size=47750KB, replication_id=1499346376, replication_reason="sync replication required to establish common bundles across all search peers"

I've also tried changing the timeout in the distsearch to but it doesn't seem to reflect correctly.

Re: New application of the search head cannot make search data

woodcock — Thu, 06 Jul 2017 03:25:31 GMT

It is all in the "how" of "I created a new application". The 2 best ways are from the CLI, tar up an existing app in the apps directory then rename the same app as something else, then untar to put the original back. Then edit the stuff in the cloned app. The next best way is from the GUI on the Search Head: Apps -> Manage apps -> Create app button, then follow the prompts. Just trash your existing one and start over.

Re: New application of the search head cannot make search data

gcusello — Thu, 06 Jul 2017 07:00:28 GMT

HI chustar,
are you sure that all the knowledge objects of your cloned app are shared at App o Global Level?
if you forgot as private some objects like eventtypes or field definitions you didn't copied them in the new app and it doesn't run.
Bye.
Giuseppe

Re: New application of the search head cannot make search data

chustar — Thu, 06 Jul 2017 13:26:23 GMT

Yeah, thsoe are being shared globally.

Re: New application of the search head cannot make search data

chustar — Thu, 06 Jul 2017 13:27:25 GMT

This was due to the replication bundle not syncing. I've updated the question.

Re: New application of the search head cannot make search data

sbbadri — Thu, 06 Jul 2017 14:22:57 GMT

To resync to can issue this command

splunk resync shcluster-replicated-config

Note: You do not need to restart the member after running this command.

Caution: This command causes an overwrite of the member's entire set of search-related configurations, resulting in the loss of any local changes.

Re: New application of the search head cannot make search data

chustar — Thu, 06 Jul 2017 22:12:23 GMT

The issue was that all of the applications on the search head were being automatically replicated to our indexer.

One of those applications had a very long file name and this caused the indexers to fail while processing the issue.

I fixed the issue by modifying the system/local/distsearch.conf on the search head to prevent that application from replicating being replicated.