Getting Data In

Write to summary index, Why are some fields are not populating?

tlmayes
Contributor

Have a query comprised of 2 subqueries (joins).  Output is exactly as expected
When I try to push that data to a summary index, only the fields from the original query make it, for all fields and event data generated from the sub queries there is nothing.    Finally, when I run the query (including '|collect index=summary' as the last line) everything expected is in the output, just not making it to the summary index.

 

 

 

index=blah_blah <followed by a search>
| join [<search string1> [ <search string 2]]
| fields _time IP DNS NETBIOS TRACKING_METHOD OS TAGS QID TITLE TYPE SEVERITY STATUS LAST_SCAN_DATETIME LAST_FOUND_DATETIME LAST_FIXED_DATETIME PUBLISHED_DATETIME THREAT_INTEL_VALUES THREAT_INTEL_IDS CVSS_V3_BASE VENDOR_REFERENCE RESULTS 
| collect index=summary

 

 

 

Output is fully populated, yet summary index is missing several fields (and the associated data).

Note: the missing fields in the summary index are all from the sub-searches/join.

 

Labels (1)
0 Karma

tlmayes
Contributor

Interesting, and thanks for the reminder (I forget about the job inspector).  No smoking gun, other than the fact it says is wrote "1,000" results??  The query returns 60,000 events.  Is there a limit to how much you can write to a summary?

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Which query? There is a limit of 50,000 events in a subsearch - could that be your issue?

0 Karma

tlmayes
Contributor

You actually provided a solution several weeks ago that resolved the query count problem for subsearches.  Sub-search#1 produces ~ 1000 events.  Outcome of sub-search#2 produces ~ 4,500 events.  The final search produces ~60,000 events (the same query that ends with "|collect index=summary"

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Are there any indications in the job inspector as to what may have happened?

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...