Knowledge Management

How to write to kvstore with java sdk?

organus
Explorer

We are trying to inject JSON directly into our KV Store instance while using a defined _key inside the JSON object.

We are not able to connect because of the following error:

2016-09-23 10:25:01 ERROR stderr:71 - com.splunk.HttpException: HTTP 400 -- <?xml version="1.0" encoding="UTF-8"?>
2016-09-23 10:25:01 ERROR stderr:71 - <response>
2016-09-23 10:25:01 ERROR stderr:71 -   <messages>
2016-09-23 10:25:01 ERROR stderr:71 -     <msg type="ERROR">Must supply 'Content-Type' header set to 'application/json'</msg>
2016-09-23 10:25:01 ERROR stderr:71 -   </messages>
2016-09-23 10:25:01 ERROR stderr:71 - </response>

We have looked for a way to add message headers to the RequestMessage object but we do not see a method in the source. Where do we inject message headers?

The method breaks on the following line:

response = service.send(kvStorePath, requestMessage);

Below is our method:

    private void writeToKVStore(File uglyJsonFile) throws IOException {
        String nextJSON = "";
        ResponseMessage response = null;
        BufferedReader br = null;
        ServiceArgs serviceArgs = null;
        Service service = null;
        RequestMessage requestMessage = null;
        int port = params.getKvStorePort();
        String kvStorePath = params.getKvStorePath();
        String kvStoreScheme = params.getKvStoreScheme();
        String kvStoreUsername = params.getkvStoreUsername();
        String kvStorePassword = params.getkvStorePassword();
        String kvStoreHost = params.getKvStoreHost();
        String kvStoreApp = params.getKvStoreApp();

        try {
            HttpService.setSslSecurityProtocol(SSLSecurityProtocol.TLSv1_2);

            serviceArgs = new ServiceArgs();
            serviceArgs.setScheme(kvStoreScheme);
            serviceArgs.setUsername(kvStoreUsername);
            serviceArgs.setPassword(kvStorePassword);
            serviceArgs.setHost(kvStoreHost);
            serviceArgs.setPort(port);
            serviceArgs.setApp(kvStoreApp); // kvStoreApp = "kvstore"


            service = Service.connect(serviceArgs);

            // TODO set up requestMessage
            // Must supply 'Content-Type' header set to 'application/json'
            requestMessage = new RequestMessage("POST");

//          service.send(params.getKvStorePath(), requestMessage);

            br = new BufferedReader(new FileReader(uglyJsonFile));

            while ((nextJSON = br.readLine()) != null) {
                requestMessage.setContent(nextJSON);
                response = service.send(kvStorePath, requestMessage);

            }

        } catch (IOException e) {
            LOGGER.error(e.getMessage());
            e.printStackTrace();
        } catch (Exception ex) {
            LOGGER.error(ex.getMessage());
            ex.printStackTrace();
            if (response != null) {
                String nextLine = "";
                LOGGER.error("Status: " + response.getStatus());
                BufferedReader responseContent = new BufferedReader(new InputStreamReader(response.getContent(), "UTF-8"));
                while((nextLine = responseContent.readLine()) != null) {
                    LOGGER.error(nextLine);
                }
            }
        } finally {
            try {
                if (br != null) {
                    br.close();
                }
            } catch (IOException e) {
                LOGGER.error(e.getMessage());
                e.printStackTrace();
                throw e;
            } 
        }

    }
0 Karma
1 Solution

organus
Explorer

We solved this problem by replicating code in the Service class. The Map returned by the RequestMessage.getHeader() method can be modified directly, without using methods in the class:

requestMessage.getHeader().put("Content-Type", "application/json");

The writeToKVStore(File) method now returns without errors. We got it all to work. This should be a fairly pluggable solution. See the working code below:

    private void writeToKVStore(File uglyJsonFile) throws IOException, Exception {
        String nextJSON = "";
        ResponseMessage response = null;
        BufferedReader br = null;
        ServiceArgs serviceArgs = null;
        Service service = null;
        RequestMessage requestMessage = null;
        int port = params.getKvStorePort();
        String kvStorePath = params.getKvStorePath();
        String kvStoreScheme = params.getKvStoreScheme();
        String kvStoreUsername = params.getkvStoreUsername();
        String kvStorePassword = params.getkvStorePassword();
        String kvStoreHost = params.getKvStoreHost();
        String kvStoreApp = params.getKvStoreApp();

        try {
            // Set up service and connect
            // This fixes an issue with conflicting security protocols in v. 1.4
            HttpService.setSslSecurityProtocol(SSLSecurityProtocol.TLSv1_2);

            serviceArgs = new ServiceArgs();
            serviceArgs.setScheme(kvStoreScheme);
            serviceArgs.setUsername(kvStoreUsername);
            serviceArgs.setPassword(kvStorePassword);
            serviceArgs.setHost(kvStoreHost);
            serviceArgs.setPort(port);
            serviceArgs.setApp(kvStoreApp); // kvStoreApp = "kvstore"

            service = Service.connect(serviceArgs);

            // Set up RequestMessage
            requestMessage = new RequestMessage("POST");
            requestMessage.getHeader().put("Content-Type", "application/json");

            // Set up stream input from file
            br = new BufferedReader(new FileReader(uglyJsonFile));

            // Loop through file and commit data to KV Store
            while ((nextJSON = br.readLine()) != null) {
                // batch_save requires an array of JSON objects.  We submit a single record at a time so we need to wrap in brackets.
                requestMessage.setContent("[" + nextJSON + "]");
                // kvStorePath = /servicesNS/nobody/kvstore/storage/collections/data/{your_collection_here}/batch_save
                response = service.send(kvStorePath, requestMessage);

            }

        } catch (IOException e) {
            LOGGER.error(e.getMessage());
            e.printStackTrace();
        } catch (Exception ex) {
            LOGGER.error(ex.getMessage());
            ex.printStackTrace();
            if (response != null) {
                String nextLine = "";
                LOGGER.error("Status: " + response.getStatus());
                BufferedReader responseContent = new BufferedReader(new InputStreamReader(response.getContent(), "UTF-8"));
                while((nextLine = responseContent.readLine()) != null) {
                    LOGGER.error(nextLine);
                }
            }
            throw ex;
        } finally {
            try {
                if (br != null) {
                    br.close();
                }
            } catch (IOException e) {
                LOGGER.error(e.getMessage());
                e.printStackTrace();
                throw e;
            } 
        }

    }

View solution in original post

organus
Explorer

We solved this problem by replicating code in the Service class. The Map returned by the RequestMessage.getHeader() method can be modified directly, without using methods in the class:

requestMessage.getHeader().put("Content-Type", "application/json");

The writeToKVStore(File) method now returns without errors. We got it all to work. This should be a fairly pluggable solution. See the working code below:

    private void writeToKVStore(File uglyJsonFile) throws IOException, Exception {
        String nextJSON = "";
        ResponseMessage response = null;
        BufferedReader br = null;
        ServiceArgs serviceArgs = null;
        Service service = null;
        RequestMessage requestMessage = null;
        int port = params.getKvStorePort();
        String kvStorePath = params.getKvStorePath();
        String kvStoreScheme = params.getKvStoreScheme();
        String kvStoreUsername = params.getkvStoreUsername();
        String kvStorePassword = params.getkvStorePassword();
        String kvStoreHost = params.getKvStoreHost();
        String kvStoreApp = params.getKvStoreApp();

        try {
            // Set up service and connect
            // This fixes an issue with conflicting security protocols in v. 1.4
            HttpService.setSslSecurityProtocol(SSLSecurityProtocol.TLSv1_2);

            serviceArgs = new ServiceArgs();
            serviceArgs.setScheme(kvStoreScheme);
            serviceArgs.setUsername(kvStoreUsername);
            serviceArgs.setPassword(kvStorePassword);
            serviceArgs.setHost(kvStoreHost);
            serviceArgs.setPort(port);
            serviceArgs.setApp(kvStoreApp); // kvStoreApp = "kvstore"

            service = Service.connect(serviceArgs);

            // Set up RequestMessage
            requestMessage = new RequestMessage("POST");
            requestMessage.getHeader().put("Content-Type", "application/json");

            // Set up stream input from file
            br = new BufferedReader(new FileReader(uglyJsonFile));

            // Loop through file and commit data to KV Store
            while ((nextJSON = br.readLine()) != null) {
                // batch_save requires an array of JSON objects.  We submit a single record at a time so we need to wrap in brackets.
                requestMessage.setContent("[" + nextJSON + "]");
                // kvStorePath = /servicesNS/nobody/kvstore/storage/collections/data/{your_collection_here}/batch_save
                response = service.send(kvStorePath, requestMessage);

            }

        } catch (IOException e) {
            LOGGER.error(e.getMessage());
            e.printStackTrace();
        } catch (Exception ex) {
            LOGGER.error(ex.getMessage());
            ex.printStackTrace();
            if (response != null) {
                String nextLine = "";
                LOGGER.error("Status: " + response.getStatus());
                BufferedReader responseContent = new BufferedReader(new InputStreamReader(response.getContent(), "UTF-8"));
                while((nextLine = responseContent.readLine()) != null) {
                    LOGGER.error(nextLine);
                }
            }
            throw ex;
        } finally {
            try {
                if (br != null) {
                    br.close();
                }
            } catch (IOException e) {
                LOGGER.error(e.getMessage());
                e.printStackTrace();
                throw e;
            } 
        }

    }
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...