Getting Data In

Write to summary index, Why are some fields are not populating?

tlmayes
Contributor

Have a query comprised of 2 subqueries (joins).  Output is exactly as expected
When I try to push that data to a summary index, only the fields from the original query make it, for all fields and event data generated from the sub queries there is nothing.    Finally, when I run the query (including '|collect index=summary' as the last line) everything expected is in the output, just not making it to the summary index.

 

 

 

index=blah_blah <followed by a search>
| join [<search string1> [ <search string 2]]
| fields _time IP DNS NETBIOS TRACKING_METHOD OS TAGS QID TITLE TYPE SEVERITY STATUS LAST_SCAN_DATETIME LAST_FOUND_DATETIME LAST_FIXED_DATETIME PUBLISHED_DATETIME THREAT_INTEL_VALUES THREAT_INTEL_IDS CVSS_V3_BASE VENDOR_REFERENCE RESULTS 
| collect index=summary

 

 

 

Output is fully populated, yet summary index is missing several fields (and the associated data).

Note: the missing fields in the summary index are all from the sub-searches/join.

 

Labels (1)
0 Karma

tlmayes
Contributor

Interesting, and thanks for the reminder (I forget about the job inspector).  No smoking gun, other than the fact it says is wrote "1,000" results??  The query returns 60,000 events.  Is there a limit to how much you can write to a summary?

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Which query? There is a limit of 50,000 events in a subsearch - could that be your issue?

0 Karma

tlmayes
Contributor

You actually provided a solution several weeks ago that resolved the query count problem for subsearches.  Sub-search#1 produces ~ 1000 events.  Outcome of sub-search#2 produces ~ 4,500 events.  The final search produces ~60,000 events (the same query that ends with "|collect index=summary"

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Are there any indications in the job inspector as to what may have happened?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...