Splunk will not extract or persist fields by default if the search parser tells it that the given fields are not referenced in the search language anywhere.
You can of course trigger the extraction by tacking on a fields command as you've discovered:
<your search> | fields someDesiredField anotherDesiredField
As far as a more general solution, you can tack on fields * to tell it to extract every field.
And at the API level, you can also send required_field_list=* on the POST when you dispatch the job.
Be aware that when you search with fields * or submit with the API arg required_field_list set to *, your searches will incur a what can be a significant performance drain, as every possible extraction will be run.
(as to why the default search UI always has every field, the presence of the FieldPicker module in the view XML triggers a change to where all searches get dispatched with the REST arg required_field_list=*. To come full circle, when you turn off 'automatic field discovery' in the UI, what's happening is it then falls back to submitting required_field_list=<just the selected fields please>)
In general if you're using the REST API you really should become as familiar with status_buckets and required_field_list sooner rather than later because behavior can be quite counterintuitive.
http://www.splunk.com/base/Documentation/4.2.1/Developer/RESTSearch
