Why does search via REST API only outputs internal debug data?

[tpaulsen]

Hi, we have a Splunk 5 system running. When we try to do a search via the REST API, we get debug output information back. When we do the search a second time, we get the results of the search. What are we doing wrong here?

Splunk SEARCH:

source="/var/opt/tomcat/logs/tomcat_access.log" fooo_group=foofoofoo | timechart span=10s avg(runtime) by fooo_tical

CODE:

    package de.fooo.jenkinsci.plugins.splunker;

    import de.fooo.jenkinsci.utils.IOMagic;
    import org.apache.commons.io.FilenameUtils;

    import java.io.File;
    import java.text.DateFormat;
    import java.text.SimpleDateFormat;
    import java.util.ArrayList;
    import java.util.Date;
    import java.util.List;

    final class SplunkSearch {

            private static final String SPLUNKRESULTDATAFILENAME = "splunkresultdata.csv";

            // 2013-02-08T11:40:58-0100
            private static final DateFormat splunkDateFormat = new SimpleDateFormat("yyyy-MM-dd'T'hh:mm:ssZ");

           private final String splunkServerUrl;
           private final String userName;
           private final String password;

           SplunkSearch(String splunkServerUrl, String userName, String password) throws Exception {
                   this.splunkServerUrl = splunkServerUrl;
                   this.userName = userName;
                   this.password = password;

                   // TODO test connection
                   // curl -k -u fkt_foofoo:splunkfooword
                   // https://fooo-splunk-foonet.de:8089/services/authentication/users
           }

           File search(String search, Date from, Date now) throws Exception {
                   File datafile;
                   if (search.startsWith("SELFTEST")) {
                           datafile = IOMagic.saveFileFromClasspath("/scripts/statistic/demodata/" + search + ".csv", search, ".csv");
                   } else {
                           System.out.println("---------------------------------------------------------------- 1");
                           List<String> command = new ArrayList<String>();
                           command.add("curl");
                           command.add("-vs");
                           command.add("-k");
                           command.add("-u");
                           command.add(userName + ":" + password);
                           command.add(splunkServerUrl);
                           command.add("-d");
                           command.add("output_mode=csv");
                           command.add("--data-urlencode");
                           command.add("search=search " + search);
                           command.add("-d");
                           command.add("earliest_time=\"" + splunkDateFormat.format(from) + "\"");
                           command.add("-d");
                           command.add("latest_time=\"" + splunkDateFormat.format(now) + "\"");
                           command.add("-d");
                           command.add("exec_mode=\"oneshot\"");
                           command.add("-o");
                           File tmpfile = File.createTempFile(FilenameUtils.getBaseName(SPLUNKRESULTDATAFILENAME),
                                           "." + FilenameUtils.getExtension(SPLUNKRESULTDATAFILENAME));
                           command.add(tmpfile.getAbsolutePath());

                           ProcessBuilder pb = new ProcessBuilder();
                           pb.command(command);
                           pb.start();
                           try {
                                   Thread.sleep(10000);
                           } catch (InterruptedException e) {
                                   e.printStackTrace();
                           }
                           // Avoid empty first line
                           datafile = File.createTempFile(FilenameUtils.getBaseName(SPLUNKRESULTDATAFILENAME),
                                           "." + FilenameUtils.getExtension(SPLUNKRESULTDATAFILENAME));
                           IOMagic.removeEmptyLines(tmpfile, datafile);
                   }
                   if (datafile.length() == 0) {
                           throw new Exception("Data file from Splunk is empty.");
                   }
                   return datafile;
           }

   }

[artemidas]

Hello, I have the same issue and it seems that this happens when the output_mode is set to 'csv' . When i issue the same request, adding the ' | table * ' command at the end of the search, I get all the information, but it is three times slower. Can you please take a look?

[tpaulsen]

Ok, our developer switched to shell and ran the following curl:

curl --get -s -k -u fkt_foofoo:foofoo4u https://foosplunknet.de:8089/servicesNS/admin/search/search/jobs/export -d output_mode=json -d exec_mode=oneshot -d earliest_time=-60m -d latest_time=now -d preview=false --data-urlencode search="search foo_group=live source="/tomcat/logs/logling.log" ltag="service/authenticate" | stats count(visitor) as visitor" |jq -r '.result | .visitor'

from the shell and the Splunk GUI and in both cases we get the same result.

Thank you,

Thomas