Getting Data In

Why does search via REST API only outputs internal debug data?

tpaulsen
Contributor

Hi, we have a Splunk 5 system running. When we try to do a search via the REST API, we get debug output information back. When we do the search a second time, we get the results of the search. What are we doing wrong here?

Splunk SEARCH:

source="/var/opt/tomcat/logs/tomcat_access.log" fooo_group=foofoofoo | timechart span=10s avg(runtime) by fooo_tical

CODE:

    package de.fooo.jenkinsci.plugins.splunker;

    import de.fooo.jenkinsci.utils.IOMagic;
    import org.apache.commons.io.FilenameUtils;

    import java.io.File;
    import java.text.DateFormat;
    import java.text.SimpleDateFormat;
    import java.util.ArrayList;
    import java.util.Date;
    import java.util.List;

    final class SplunkSearch {

            private static final String SPLUNKRESULTDATAFILENAME = "splunkresultdata.csv";

            // 2013-02-08T11:40:58-0100
            private static final DateFormat splunkDateFormat = new SimpleDateFormat("yyyy-MM-dd'T'hh:mm:ssZ");

           private final String splunkServerUrl;
           private final String userName;
           private final String password;

           SplunkSearch(String splunkServerUrl, String userName, String password) throws Exception {
                   this.splunkServerUrl = splunkServerUrl;
                   this.userName = userName;
                   this.password = password;

                   // TODO test connection
                   // curl -k -u fkt_foofoo:splunkfooword
                   // https://fooo-splunk-foonet.de:8089/services/authentication/users
           }

           File search(String search, Date from, Date now) throws Exception {
                   File datafile;
                   if (search.startsWith("SELFTEST")) {
                           datafile = IOMagic.saveFileFromClasspath("/scripts/statistic/demodata/" + search + ".csv", search, ".csv");
                   } else {
                           System.out.println("---------------------------------------------------------------- 1");
                           List<String> command = new ArrayList<String>();
                           command.add("curl");
                           command.add("-vs");
                           command.add("-k");
                           command.add("-u");
                           command.add(userName + ":" + password);
                           command.add(splunkServerUrl);
                           command.add("-d");
                           command.add("output_mode=csv");
                           command.add("--data-urlencode");
                           command.add("search=search " + search);
                           command.add("-d");
                           command.add("earliest_time=\"" + splunkDateFormat.format(from) + "\"");
                           command.add("-d");
                           command.add("latest_time=\"" + splunkDateFormat.format(now) + "\"");
                           command.add("-d");
                           command.add("exec_mode=\"oneshot\"");
                           command.add("-o");
                           File tmpfile = File.createTempFile(FilenameUtils.getBaseName(SPLUNKRESULTDATAFILENAME),
                                           "." + FilenameUtils.getExtension(SPLUNKRESULTDATAFILENAME));
                           command.add(tmpfile.getAbsolutePath());

                           ProcessBuilder pb = new ProcessBuilder();
                           pb.command(command);
                           pb.start();
                           try {
                                   Thread.sleep(10000);
                           } catch (InterruptedException e) {
                                   e.printStackTrace();
                           }
                           // Avoid empty first line
                           datafile = File.createTempFile(FilenameUtils.getBaseName(SPLUNKRESULTDATAFILENAME),
                                           "." + FilenameUtils.getExtension(SPLUNKRESULTDATAFILENAME));
                           IOMagic.removeEmptyLines(tmpfile, datafile);
                   }
                   if (datafile.length() == 0) {
                           throw new Exception("Data file from Splunk is empty.");
                   }
                   return datafile;
           }

   }
Tags (3)
0 Karma

artemidas
New Member

Hello, I have the same issue and it seems that this happens when the output_mode is set to 'csv' . When i issue the same request, adding the ' | table * ' command at the end of the search, I get all the information, but it is three times slower. Can you please take a look?

0 Karma

tpaulsen
Contributor

Ok, our developer switched to shell and ran the following curl:

curl --get -s -k -u fkt_foofoo:foofoo4u https://foosplunknet.de:8089/servicesNS/admin/search/search/jobs/export -d output_mode=json -d exec_mode=oneshot -d earliest_time=-60m -d latest_time=now -d preview=false --data-urlencode search="search foo_group=live source="/tomcat/logs/logling.log" ltag="service/authenticate" | stats count(visitor) as visitor" |jq -r '.result | .visitor'

from the shell and the Splunk GUI and in both cases we get the same result.

Thank you,

Thomas

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...