topic Re: Is collect command working correctly ? in Knowledge Management

Is collect command working correctly ?

dreadangel — Wed, 30 Sep 2020 00:19:56 GMT

index=* sourcetype="..."  ... | eval new_field="new_value-".old_field, new_field_id="[some new id]".old_field_id | table *  | collect index=inx_copy_data

Straightforward task - to select events, filter them, add some new fields, copy the results to another index.
But in inx_copy_data not all data is copied - I can't find the new fields (new_field, new_field_id) nor the old ones (old_field, old_field_id).
Do I miss something?

Re: Is collect command working correctly ?

DavidHourani — Thu, 02 May 2019 12:45:02 GMT

Hi @dreadangel

Use stats instead of table * because using tabledoesn't transform the results and collect would still get the raw data.

something like this should do the trick :

index=* sourcetype="..."  ... | eval new_field="new_value-".old_field, new_field_id="[some new id]".old_field_id | stats values(*) as * by _time  | collect index=inx_copy_data

Reference here :
https://docs.splunk.com/Documentation/Splunk/7.2.6/SearchReference/Collect

Note:
Summary indexing counts against ur license so careful when using it.

Cheers,
David

Re: Is collect command working correctly ?

dreadangel — Thu, 02 May 2019 13:18:42 GMT

thank you!!

Re: Is collect command working correctly ?

dreadangel — Tue, 07 May 2019 08:54:46 GMT

so the big problem is due to _raw field, correct? Is any way to avoid this? Or I should omit _raw fields from collect command ?

Re: Is collect command working correctly ?

DavidHourani — Tue, 07 May 2019 10:30:19 GMT

yeah, sure you can omit it to avoid having the same info multiple times. It's actually best to replace that * with the exact fields you need