This is sent to devinfo also :
We are using the Splunk javasdk to query out of a splunk
alarms index to put alarm into and oracle table and to our
alarm system. It runs every minute.
The query is working 99% of the time, but we have some
unusual minutes where the count of alarm events is usually
high, as much as 5000 events, never close to the SDK limit of 50,000 .
When this happend the query fails, and gives a BOGUS
error message.
2018-01-11T23:58:00.046Z, level=ERROR, host=njgpdwi02, ssys=ETL, tid=214, cat=npac.splunk_etl.ETL_common, gid=null, text="ETL_splunk_TX: Exception while processing splunk ETL: (Connection refused (Connection refused))." bldi=default-value-from-handler
We know the "Connection refused" is bogus because curl REST command still connect to the same port
AND moving the splunk query to a minute with fewer events, typically fewer than 20 and
the splunk query connects and works.
It seem clear the large number of events in some single minutes are the issue
but we get a bogus message from the Splunk Java SDK.
The Java SDK One shot query to splunk is set up like this;
Args osArgs = new Args();
osArgs.put("earliest_time", sdf.format(ts1)); // get all alarms posted since the earliest time
osArgs.put("count",fetchLimit); // set to zero to fetch all alarms (up to 50000 limit imposed by the API)
osArgs.put("output_mode","xml");
String osQuery = "search index=npac_alarm | reverse ";
The count is set to 60, so bundles of 60 events should come.
While the count of events that trigger the problem is relatively low at a few
thousand, some of these include java stack traces and can be quite
large both individually, and in sum.
We are looking for guidance to clear the problem. I read that
using export queries from the SDK is preferable if there are a big number of
events. Could there be a buffer we are overflowing? Is count = 60 too low?
Help appreciated. We are a support customer but I have not opened a ticket
my understadnig is writing to dev-info is the correct method.
