How to write to kvstore with java sdk?

organus — Fri, 23 Sep 2016 17:21:27 GMT

We are trying to inject JSON directly into our KV Store instance while using a defined _key inside the JSON object.

We are not able to connect because of the following error:

2016-09-23 10:25:01 ERROR stderr:71 - com.splunk.HttpException: HTTP 400 -- <?xml version="1.0" encoding="UTF-8"?>
2016-09-23 10:25:01 ERROR stderr:71 - <response>
2016-09-23 10:25:01 ERROR stderr:71 -   <messages>
2016-09-23 10:25:01 ERROR stderr:71 -     <msg type="ERROR">Must supply 'Content-Type' header set to 'application/json'</msg>
2016-09-23 10:25:01 ERROR stderr:71 -   </messages>
2016-09-23 10:25:01 ERROR stderr:71 - </response>

We have looked for a way to add message headers to the RequestMessage object but we do not see a method in the source. Where do we inject message headers?

The method breaks on the following line:

response = service.send(kvStorePath, requestMessage);

Below is our method:

    private void writeToKVStore(File uglyJsonFile) throws IOException {
        String nextJSON = "";
        ResponseMessage response = null;
        BufferedReader br = null;
        ServiceArgs serviceArgs = null;
        Service service = null;
        RequestMessage requestMessage = null;
        int port = params.getKvStorePort();
        String kvStorePath = params.getKvStorePath();
        String kvStoreScheme = params.getKvStoreScheme();
        String kvStoreUsername = params.getkvStoreUsername();
        String kvStorePassword = params.getkvStorePassword();
        String kvStoreHost = params.getKvStoreHost();
        String kvStoreApp = params.getKvStoreApp();

        try {
            HttpService.setSslSecurityProtocol(SSLSecurityProtocol.TLSv1_2);

            serviceArgs = new ServiceArgs();
            serviceArgs.setScheme(kvStoreScheme);
            serviceArgs.setUsername(kvStoreUsername);
            serviceArgs.setPassword(kvStorePassword);
            serviceArgs.setHost(kvStoreHost);
            serviceArgs.setPort(port);
            serviceArgs.setApp(kvStoreApp); // kvStoreApp = "kvstore"


            service = Service.connect(serviceArgs);

            // TODO set up requestMessage
            // Must supply 'Content-Type' header set to 'application/json'
            requestMessage = new RequestMessage("POST");

//          service.send(params.getKvStorePath(), requestMessage);

            br = new BufferedReader(new FileReader(uglyJsonFile));

            while ((nextJSON = br.readLine()) != null) {
                requestMessage.setContent(nextJSON);
                response = service.send(kvStorePath, requestMessage);

            }

        } catch (IOException e) {
            LOGGER.error(e.getMessage());
            e.printStackTrace();
        } catch (Exception ex) {
            LOGGER.error(ex.getMessage());
            ex.printStackTrace();
            if (response != null) {
                String nextLine = "";
                LOGGER.error("Status: " + response.getStatus());
                BufferedReader responseContent = new BufferedReader(new InputStreamReader(response.getContent(), "UTF-8"));
                while((nextLine = responseContent.readLine()) != null) {
                    LOGGER.error(nextLine);
                }
            }
        } finally {
            try {
                if (br != null) {
                    br.close();
                }
            } catch (IOException e) {
                LOGGER.error(e.getMessage());
                e.printStackTrace();
                throw e;
            } 
        }

    }

Re: How to write to kvstore with java sdk?

organus — Fri, 23 Sep 2016 19:24:01 GMT

We solved this problem by replicating code in the Service class. The Map returned by the RequestMessage.getHeader() method can be modified directly, without using methods in the class:

requestMessage.getHeader().put("Content-Type", "application/json");

The writeToKVStore(File) method now returns without errors. We got it all to work. This should be a fairly pluggable solution. See the working code below:

    private void writeToKVStore(File uglyJsonFile) throws IOException, Exception {
        String nextJSON = "";
        ResponseMessage response = null;
        BufferedReader br = null;
        ServiceArgs serviceArgs = null;
        Service service = null;
        RequestMessage requestMessage = null;
        int port = params.getKvStorePort();
        String kvStorePath = params.getKvStorePath();
        String kvStoreScheme = params.getKvStoreScheme();
        String kvStoreUsername = params.getkvStoreUsername();
        String kvStorePassword = params.getkvStorePassword();
        String kvStoreHost = params.getKvStoreHost();
        String kvStoreApp = params.getKvStoreApp();

        try {
            // Set up service and connect
            // This fixes an issue with conflicting security protocols in v. 1.4
            HttpService.setSslSecurityProtocol(SSLSecurityProtocol.TLSv1_2);

            serviceArgs = new ServiceArgs();
            serviceArgs.setScheme(kvStoreScheme);
            serviceArgs.setUsername(kvStoreUsername);
            serviceArgs.setPassword(kvStorePassword);
            serviceArgs.setHost(kvStoreHost);
            serviceArgs.setPort(port);
            serviceArgs.setApp(kvStoreApp); // kvStoreApp = "kvstore"

            service = Service.connect(serviceArgs);

            // Set up RequestMessage
            requestMessage = new RequestMessage("POST");
            requestMessage.getHeader().put("Content-Type", "application/json");

            // Set up stream input from file
            br = new BufferedReader(new FileReader(uglyJsonFile));

            // Loop through file and commit data to KV Store
            while ((nextJSON = br.readLine()) != null) {
                // batch_save requires an array of JSON objects.  We submit a single record at a time so we need to wrap in brackets.
                requestMessage.setContent("[" + nextJSON + "]");
                // kvStorePath = /servicesNS/nobody/kvstore/storage/collections/data/{your_collection_here}/batch_save
                response = service.send(kvStorePath, requestMessage);

            }

        } catch (IOException e) {
            LOGGER.error(e.getMessage());
            e.printStackTrace();
        } catch (Exception ex) {
            LOGGER.error(ex.getMessage());
            ex.printStackTrace();
            if (response != null) {
                String nextLine = "";
                LOGGER.error("Status: " + response.getStatus());
                BufferedReader responseContent = new BufferedReader(new InputStreamReader(response.getContent(), "UTF-8"));
                while((nextLine = responseContent.readLine()) != null) {
                    LOGGER.error(nextLine);
                }
            }
            throw ex;
        } finally {
            try {
                if (br != null) {
                    br.close();
                }
            } catch (IOException e) {
                LOGGER.error(e.getMessage());
                e.printStackTrace();
                throw e;
            } 
        }

    }

topic How to write to kvstore with java sdk? in Knowledge Management

How to write to kvstore with java sdk?

Re: How to write to kvstore with java sdk?