topic Re: Write to summary index, Why are some fields are not populating? in Getting Data In

Write to summary index, Why are some fields are not populating?

tlmayes — Fri, 15 Jul 2022 17:25:14 GMT

Have a query comprised of 2 subqueries (joins). Output is exactly as expected.
When I try to push that data to a summary index, only the fields from the original query make it, for all fields and event data generated from the sub queries there is nothing. Finally, when I run the query (including '|collect index=summary' as the last line) everything expected is in the output, just not making it to the summary index.

index=blah_blah <followed by a search> | join [<search string1> [ <search string 2]] | fields _time IP DNS NETBIOS TRACKING_METHOD OS TAGS QID TITLE TYPE SEVERITY STATUS LAST_SCAN_DATETIME LAST_FOUND_DATETIME LAST_FIXED_DATETIME PUBLISHED_DATETIME THREAT_INTEL_VALUES THREAT_INTEL_IDS CVSS_V3_BASE VENDOR_REFERENCE RESULTS | collect index=summary

Output is fully populated, yet summary index is missing several fields (and the associated data).

Note: the missing fields in the summary index are all from the sub-searches/join.

Re: Write to summary index, Why are some fields are not populating?

ITWhisperer — Fri, 15 Jul 2022 17:52:06 GMT

Are there any indications in the job inspector as to what may have happened?

Re: Write to summary index, Why are some fields are not populating?

tlmayes — Fri, 15 Jul 2022 17:58:39 GMT

Interesting, and thanks for the reminder (I forget about the job inspector). No smoking gun, other than the fact it says is wrote "1,000" results?? The query returns 60,000 events. Is there a limit to how much you can write to a summary?

Re: Write to summary index, Why are some fields are not populating?

ITWhisperer — Fri, 15 Jul 2022 18:02:57 GMT

Which query? There is a limit of 50,000 events in a subsearch - could that be your issue?

Re: Write to summary index, Why are some fields are not populating?

tlmayes — Fri, 15 Jul 2022 18:10:25 GMT

You actually provided a solution several weeks ago that resolved the query count problem for subsearches. Sub-search#1 produces ~ 1000 events. Outcome of sub-search#2 produces ~ 4,500 events. The final search produces ~60,000 events (the same query that ends with "|collect index=summary"