topic Re: collect command wouldn't save to an index in Getting Data In

collect command wouldn't save to an index

sintjm — Thu, 11 Jul 2024 10:26:13 GMT

i have a search in my query where i spool data from an API but then the collect command does not allow me to save the search into my index. Any ideas?

Re: collect command wouldn't save to an index

ITWhisperer — Thu, 11 Jul 2024 10:27:50 GMT

"does not allow" is a bit vague - what errors do you get? what shows up in the log? what other information do you have?

Re: collect command wouldn't save to an index

gcusello — Thu, 11 Jul 2024 10:29:08 GMT

Hi @sintjm,

I suppose that you created the index and it has the correct grants.

did you explicit the index name in the search?

<your_search> | table ............ | collect index=your_index

Ciao.

Giuseppe

Re: collect command wouldn't save to an index

sintjm — Thu, 11 Jul 2024 10:37:20 GMT

yes please, I did as such

Re: collect command wouldn't save to an index

sintjm — Thu, 11 Jul 2024 10:38:40 GMT

There was no logs showing after i searched the index, sorry

Re: collect command wouldn't save to an index

PickleRick — Thu, 11 Jul 2024 10:42:53 GMT

There's no point doing spath if in the next step you leave only the original data field. But that's beside the point.

First step in debugging this would be to remove the collect command from your pipeline and see what the results look like.

Re: collect command wouldn't save to an index

ITWhisperer — Thu, 11 Jul 2024 10:42:57 GMT

You appear to be doing spath on data, then only keeping data which makes the spath redundant! What timestamps are in the events returned by the curl command?

Re: collect command wouldn't save to an index

ITWhisperer — Thu, 11 Jul 2024 10:43:51 GMT

What about the logs after the collect?

Re: collect command wouldn't save to an index

sintjm — Thu, 11 Jul 2024 10:52:06 GMT

Re: collect command wouldn't save to an index

sintjm — Thu, 11 Jul 2024 10:59:57 GMT

so there is a field in the log named data and that's where i need my log from. I also removed spath command from the query. Removing the collect command still shows the log curled from the endpoint.

Re: collect command wouldn't save to an index

sintjm — Thu, 11 Jul 2024 11:02:41 GMT

Because i don't see the events in from searching the index I created, I can't tell the timestamp. I added

| eval _time=now()

to the query so that it would put the recent time when the endpoint was reached

Re: collect command wouldn't save to an index

ITWhisperer — Thu, 11 Jul 2024 11:35:19 GMT

So, with the spath removed and the _time field added, do you events now show up in the index (after the collect command)? If not, what does the search log say for the search with the collect command?

Re: collect command wouldn't save to an index

ITWhisperer — Thu, 11 Jul 2024 11:36:27 GMT

Another possibility is to try

| table data

instead of fields

Re: collect command wouldn't save to an index

sintjm — Thu, 11 Jul 2024 14:11:31 GMT

I tried using

| table data

but there is still no data when searching the index.

With the spath removed and the _time field added there are still no events in the index and the search log can be seen below. There is still data coming through the search.

Re: collect command wouldn't save to an index

PickleRick — Thu, 11 Jul 2024 12:38:32 GMT

This is not the search log. You get a search log when you click the "Job" button and pick "Inspect Job". On top of the screen you'll have a link to search log.

Re: collect command wouldn't save to an index

sintjm — Fri, 12 Jul 2024 08:31:52 GMT

The search log