topic Re: Structured fields and values in json api results in Getting Data In

Structured fields and values in json api results

stevesq — Wed, 18 May 2011 05:52:09 GMT

In your REST API documentation you have the following json example:

    // sample JSON output
// https://localhost:8089/services/search/jobs/1234/results?output_mode=json

[
{
"_cd": "0:4374557",
"_index": "main",
"_kv": "1",
"_meta": " date_second::36 date_hour::19 date_minute::11 date_year::2008 date_month::january date_mday::21 date_wday::monday date_zone::-480 punct::_[//:::_-]____\\\"@...\\\"...",
"_raw": "I [21/Jan/2008:19:11:36 -0800] Added remote printer \"HPLaserJ@10.1.1.123\"...",
"_serial": "0",
"_time": "1200971496",
"date_hour": "19",
"date_mday": "21",
"date_minute": "11",
"date_month": "january",
"date_second": "36",
"date_wday": "monday",
"date_year": "2008",
"date_zone": "-480",
"host": "decider.local",
"linecount": "1",
"punct": "_[//:::_-]____\"@...\"...",
"source": "/var/log/cups/error_log",
"sourcetype": "cups_error"
},

However when I make an api request and then check results from the /results endpoint, I only see the raw data and a few other fields, like so:

{
"_cd": "6:719660",
"_indextime": "1305696946",
"_raw": "id=\"123\" color=\"red\" foo= model=\"accord\"",
"_serial": "93",
"_si": "log.example.com\nmain",
"_sourcetype": "car_sourcetype",
"_time": "2010-07-18T19:19:30.000+00:00",
"host": "log.example.com",
"index": "main",
"linecount": "1",
"source": "cars",
"sourcetype": "cars_sourcetype",
"splunk_server": "log.example.com"

Note that _kv is missing, as well as the broken-out keys and values. What triggers these to be put in the api results? I can't find anything obvious in the api docs.

-Steve

Re: Structured fields and values in json api results

hazekamp — Wed, 18 May 2011 19:02:31 GMT

stevesq,

I suspect key-value pair extracting is not occurring for your sourcetype. Key-value extraction is configured via props.conf/transforms.conf. Once key-value pair extractions are specified for your data, search results will reflect this information and so will results retrieved via the api.

What's interesting is that Splunk is not automatically extracting key=value pairs in your _raw message. Perhaps KV_MODE is set to None or "Field discovery" was disabled (if search was executed from the UI).

Can you verify flashtimeline searches return the expected "id/color/foo/model" fields?

Re: Structured fields and values in json api results

stevesq — Wed, 18 May 2011 20:50:23 GMT

"search results will reflect this information and so will results retrieved via the api."

My search results seem to be keyed fine - e.g. if I pipe a search to "table *" or search for "color=red", the right things happen. Splunk is clearly treating my data as properly separated-out rows/cols/cells.

Re: Structured fields and values in json api results

stevesq — Sat, 21 May 2011 14:41:46 GMT

I made it work by including

required_field_list=*

in the search POST:

curl -u admin:foobar -k https://localhost:8089/services/search/jobs -d"search=search source=cars | head 1 &required_field_list=*"

Could be a bug.