Getting Data In

Why does search via REST API only outputs internal debug data?

Contributor

Hi, we have a Splunk 5 system running. When we try to do a search via the REST API, we get debug output information back. When we do the search a second time, we get the results of the search. What are we doing wrong here?

Splunk SEARCH:

source="/var/opt/tomcat/logs/tomcat_access.log" fooo_group=foofoofoo | timechart span=10s avg(runtime) by fooo_tical

CODE:

    package de.fooo.jenkinsci.plugins.splunker;

    import de.fooo.jenkinsci.utils.IOMagic;
    import org.apache.commons.io.FilenameUtils;

    import java.io.File;
    import java.text.DateFormat;
    import java.text.SimpleDateFormat;
    import java.util.ArrayList;
    import java.util.Date;
    import java.util.List;

    final class SplunkSearch {

            private static final String SPLUNKRESULTDATAFILENAME = "splunkresultdata.csv";

            // 2013-02-08T11:40:58-0100
            private static final DateFormat splunkDateFormat = new SimpleDateFormat("yyyy-MM-dd'T'hh:mm:ssZ");

           private final String splunkServerUrl;
           private final String userName;
           private final String password;

           SplunkSearch(String splunkServerUrl, String userName, String password) throws Exception {
                   this.splunkServerUrl = splunkServerUrl;
                   this.userName = userName;
                   this.password = password;

                   // TODO test connection
                   // curl -k -u fkt_foofoo:splunkfooword
                   // https://fooo-splunk-foonet.de:8089/services/authentication/users
           }

           File search(String search, Date from, Date now) throws Exception {
                   File datafile;
                   if (search.startsWith("SELFTEST")) {
                           datafile = IOMagic.saveFileFromClasspath("/scripts/statistic/demodata/" + search + ".csv", search, ".csv");
                   } else {
                           System.out.println("---------------------------------------------------------------- 1");
                           List<String> command = new ArrayList<String>();
                           command.add("curl");
                           command.add("-vs");
                           command.add("-k");
                           command.add("-u");
                           command.add(userName + ":" + password);
                           command.add(splunkServerUrl);
                           command.add("-d");
                           command.add("output_mode=csv");
                           command.add("--data-urlencode");
                           command.add("search=search " + search);
                           command.add("-d");
                           command.add("earliest_time=\"" + splunkDateFormat.format(from) + "\"");
                           command.add("-d");
                           command.add("latest_time=\"" + splunkDateFormat.format(now) + "\"");
                           command.add("-d");
                           command.add("exec_mode=\"oneshot\"");
                           command.add("-o");
                           File tmpfile = File.createTempFile(FilenameUtils.getBaseName(SPLUNKRESULTDATAFILENAME),
                                           "." + FilenameUtils.getExtension(SPLUNKRESULTDATAFILENAME));
                           command.add(tmpfile.getAbsolutePath());

                           ProcessBuilder pb = new ProcessBuilder();
                           pb.command(command);
                           pb.start();
                           try {
                                   Thread.sleep(10000);
                           } catch (InterruptedException e) {
                                   e.printStackTrace();
                           }
                           // Avoid empty first line
                           datafile = File.createTempFile(FilenameUtils.getBaseName(SPLUNKRESULTDATAFILENAME),
                                           "." + FilenameUtils.getExtension(SPLUNKRESULTDATAFILENAME));
                           IOMagic.removeEmptyLines(tmpfile, datafile);
                   }
                   if (datafile.length() == 0) {
                           throw new Exception("Data file from Splunk is empty.");
                   }
                   return datafile;
           }

   }
Tags (3)
0 Karma

New Member

Hello, I have the same issue and it seems that this happens when the output_mode is set to 'csv' . When i issue the same request, adding the ' | table * ' command at the end of the search, I get all the information, but it is three times slower. Can you please take a look?

0 Karma

Contributor

Ok, our developer switched to shell and ran the following curl:

curl --get -s -k -u fkt_foofoo:foofoo4u https://foosplunknet.de:8089/servicesNS/admin/search/search/jobs/export -d output_mode=json -d exec_mode=oneshot -d earliest_time=-60m -d latest_time=now -d preview=false --data-urlencode search="search foo_group=live source="/tomcat/logs/logling.log" ltag="service/authenticate" | stats count(visitor) as visitor" |jq -r '.result | .visitor'

from the shell and the Splunk GUI and in both cases we get the same result.

Thank you,

Thomas

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!