We know the "Connection refused" is bogus because curl REST command still connect to the same port
AND moving the splunk query to a minute with fewer events, typically fewer than 20 and
the splunk query connects and works.
It seem clear the large number of events in some single minutes are the issue
but we get a bogus message from the Splunk Java SDK.
The Java SDK One shot query to splunk is set up like this;
Args osArgs = new Args();
osArgs.put("earliesttime", sdf.format(ts1)); // get all alarms posted since the earliest time
osArgs.put("count",fetchLimit); // set to zero to fetch all alarms (up to 50000 limit imposed by the API)
String osQuery = "search index=npac_alarm | reverse ";
The count is set to 60, so bundles of 60 events should come.
While the count of events that trigger the problem is relatively low at a few
thousand, some of these include java stack traces and can be quite
large both individually, and in sum.
We are looking for guidance to clear the problem. I read that
using export queries from the SDK is preferable if there are a big number of
events. Could there be a buffer we are overflowing? Is count = 60 too low?
Help appreciated. We are a support customer but I have not opened a ticket
my understadnig is writing to dev-info is the correct method.