i am not recieving any xml for /services/search/jobs/
... but i get xml for /services/search/jobs/
I'm not sure i entirely understand the difference between the two... but this endpoint doesn't seem to work for ANY search.
Yeah, I confused by this too the first time I tried to search via the REST API.
I think it's easier to understand based on the new UI in Splunk 4 than it was in 3.x.
If you run a search from the UI, there are three different tabs you can switch between. The first two show you your "events" ("Event list", and "Event table" modes), and the final tab is the "Results table".
So basically your "events" are untransformed events. This includes the
_raw fields. Generally this is anything that happens before a
chart search command. The "results" are only available if you do some sort of transformation and would only include the fields explicitly listed in your
stats command, for example.
To make matters worse, it seems that sometime the terms "events" and "results" are used interchangeably.
If you are running 4.1, you can use the 'Job Inspector' to see the different between these 2. Look for the entry
eventSearch vs the
reportSearch. There is also field called
search that shows the combination of both of these, and I think will always match your actual search string.
Another gotcha here is that if you are doing a transformation on your events and running the search from the UI than you can see both the events and the results. I don't think it works this way by default for REST searches. Try setting
status_buckets=300 when you submit your search job. (This is similar to the way a saved search runs vs running a search interactively.... It's more efficient for splunk NOT to keep both event and results when it doesn't have to. There can be a big savings by simply not extracting all the fields when splunk knows their aren't going to be used.)
Thank you. This helps a lot. It still raises my eyebrow... but it points me in a good direction. Thanks Lowell.
It may also help to poke around in the
$SPLUNK_HOME/var/run/splunk/dispatch/<jobid> folders. There you should can see what splunk is actually storing for a given search job. For example, you'll see the results in
results.csv.gz and your events in the
events folder along with other files that correspond to the various endpoints. (Of course just browsing the endpoints with a browser can be helpful too.)