Knowledge Management
Highlighted

How to write to kvstore with java sdk?

Explorer

We are trying to inject JSON directly into our KV Store instance while using a defined _key inside the JSON object.

We are not able to connect because of the following error:

2016-09-23 10:25:01 ERROR stderr:71 - com.splunk.HttpException: HTTP 400 -- <?xml version="1.0" encoding="UTF-8"?>
2016-09-23 10:25:01 ERROR stderr:71 - <response>
2016-09-23 10:25:01 ERROR stderr:71 -   <messages>
2016-09-23 10:25:01 ERROR stderr:71 -     <msg type="ERROR">Must supply 'Content-Type' header set to 'application/json'</msg>
2016-09-23 10:25:01 ERROR stderr:71 -   </messages>
2016-09-23 10:25:01 ERROR stderr:71 - </response>

We have looked for a way to add message headers to the RequestMessage object but we do not see a method in the source. Where do we inject message headers?

The method breaks on the following line:

response = service.send(kvStorePath, requestMessage);

Below is our method:

    private void writeToKVStore(File uglyJsonFile) throws IOException {
        String nextJSON = "";
        ResponseMessage response = null;
        BufferedReader br = null;
        ServiceArgs serviceArgs = null;
        Service service = null;
        RequestMessage requestMessage = null;
        int port = params.getKvStorePort();
        String kvStorePath = params.getKvStorePath();
        String kvStoreScheme = params.getKvStoreScheme();
        String kvStoreUsername = params.getkvStoreUsername();
        String kvStorePassword = params.getkvStorePassword();
        String kvStoreHost = params.getKvStoreHost();
        String kvStoreApp = params.getKvStoreApp();

        try {
            HttpService.setSslSecurityProtocol(SSLSecurityProtocol.TLSv1_2);

            serviceArgs = new ServiceArgs();
            serviceArgs.setScheme(kvStoreScheme);
            serviceArgs.setUsername(kvStoreUsername);
            serviceArgs.setPassword(kvStorePassword);
            serviceArgs.setHost(kvStoreHost);
            serviceArgs.setPort(port);
            serviceArgs.setApp(kvStoreApp); // kvStoreApp = "kvstore"


            service = Service.connect(serviceArgs);

            // TODO set up requestMessage
            // Must supply 'Content-Type' header set to 'application/json'
            requestMessage = new RequestMessage("POST");

//          service.send(params.getKvStorePath(), requestMessage);

            br = new BufferedReader(new FileReader(uglyJsonFile));

            while ((nextJSON = br.readLine()) != null) {
                requestMessage.setContent(nextJSON);
                response = service.send(kvStorePath, requestMessage);

            }

        } catch (IOException e) {
            LOGGER.error(e.getMessage());
            e.printStackTrace();
        } catch (Exception ex) {
            LOGGER.error(ex.getMessage());
            ex.printStackTrace();
            if (response != null) {
                String nextLine = "";
                LOGGER.error("Status: " + response.getStatus());
                BufferedReader responseContent = new BufferedReader(new InputStreamReader(response.getContent(), "UTF-8"));
                while((nextLine = responseContent.readLine()) != null) {
                    LOGGER.error(nextLine);
                }
            }
        } finally {
            try {
                if (br != null) {
                    br.close();
                }
            } catch (IOException e) {
                LOGGER.error(e.getMessage());
                e.printStackTrace();
                throw e;
            } 
        }

    }
0 Karma
Highlighted

Re: How to write to kvstore with java sdk?

Explorer

We solved this problem by replicating code in the Service class. The Map returned by the RequestMessage.getHeader() method can be modified directly, without using methods in the class:

requestMessage.getHeader().put("Content-Type", "application/json");

The writeToKVStore(File) method now returns without errors. We got it all to work. This should be a fairly pluggable solution. See the working code below:

    private void writeToKVStore(File uglyJsonFile) throws IOException, Exception {
        String nextJSON = "";
        ResponseMessage response = null;
        BufferedReader br = null;
        ServiceArgs serviceArgs = null;
        Service service = null;
        RequestMessage requestMessage = null;
        int port = params.getKvStorePort();
        String kvStorePath = params.getKvStorePath();
        String kvStoreScheme = params.getKvStoreScheme();
        String kvStoreUsername = params.getkvStoreUsername();
        String kvStorePassword = params.getkvStorePassword();
        String kvStoreHost = params.getKvStoreHost();
        String kvStoreApp = params.getKvStoreApp();

        try {
            // Set up service and connect
            // This fixes an issue with conflicting security protocols in v. 1.4
            HttpService.setSslSecurityProtocol(SSLSecurityProtocol.TLSv1_2);

            serviceArgs = new ServiceArgs();
            serviceArgs.setScheme(kvStoreScheme);
            serviceArgs.setUsername(kvStoreUsername);
            serviceArgs.setPassword(kvStorePassword);
            serviceArgs.setHost(kvStoreHost);
            serviceArgs.setPort(port);
            serviceArgs.setApp(kvStoreApp); // kvStoreApp = "kvstore"

            service = Service.connect(serviceArgs);

            // Set up RequestMessage
            requestMessage = new RequestMessage("POST");
            requestMessage.getHeader().put("Content-Type", "application/json");

            // Set up stream input from file
            br = new BufferedReader(new FileReader(uglyJsonFile));

            // Loop through file and commit data to KV Store
            while ((nextJSON = br.readLine()) != null) {
                // batch_save requires an array of JSON objects.  We submit a single record at a time so we need to wrap in brackets.
                requestMessage.setContent("[" + nextJSON + "]");
                // kvStorePath = /servicesNS/nobody/kvstore/storage/collections/data/{your_collection_here}/batch_save
                response = service.send(kvStorePath, requestMessage);

            }

        } catch (IOException e) {
            LOGGER.error(e.getMessage());
            e.printStackTrace();
        } catch (Exception ex) {
            LOGGER.error(ex.getMessage());
            ex.printStackTrace();
            if (response != null) {
                String nextLine = "";
                LOGGER.error("Status: " + response.getStatus());
                BufferedReader responseContent = new BufferedReader(new InputStreamReader(response.getContent(), "UTF-8"));
                while((nextLine = responseContent.readLine()) != null) {
                    LOGGER.error(nextLine);
                }
            }
            throw ex;
        } finally {
            try {
                if (br != null) {
                    br.close();
                }
            } catch (IOException e) {
                LOGGER.error(e.getMessage());
                e.printStackTrace();
                throw e;
            } 
        }

    }

View solution in original post