Getting Data In

Write to summary index, Why are some fields are not populating?

tlmayes
Contributor

Have a query comprised of 2 subqueries (joins).  Output is exactly as expected
When I try to push that data to a summary index, only the fields from the original query make it, for all fields and event data generated from the sub queries there is nothing.    Finally, when I run the query (including '|collect index=summary' as the last line) everything expected is in the output, just not making it to the summary index.

 

 

 

index=blah_blah <followed by a search>
| join [<search string1> [ <search string 2]]
| fields _time IP DNS NETBIOS TRACKING_METHOD OS TAGS QID TITLE TYPE SEVERITY STATUS LAST_SCAN_DATETIME LAST_FOUND_DATETIME LAST_FIXED_DATETIME PUBLISHED_DATETIME THREAT_INTEL_VALUES THREAT_INTEL_IDS CVSS_V3_BASE VENDOR_REFERENCE RESULTS 
| collect index=summary

 

 

 

Output is fully populated, yet summary index is missing several fields (and the associated data).

Note: the missing fields in the summary index are all from the sub-searches/join.

 

0 Karma

tlmayes
Contributor

Interesting, and thanks for the reminder (I forget about the job inspector).  No smoking gun, other than the fact it says is wrote "1,000" results??  The query returns 60,000 events.  Is there a limit to how much you can write to a summary?

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Which query? There is a limit of 50,000 events in a subsearch - could that be your issue?

0 Karma

tlmayes
Contributor

You actually provided a solution several weeks ago that resolved the query count problem for subsearches.  Sub-search#1 produces ~ 1000 events.  Outcome of sub-search#2 produces ~ 4,500 events.  The final search produces ~60,000 events (the same query that ends with "|collect index=summary"

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Are there any indications in the job inspector as to what may have happened?

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Where Innovation Takes Flight: The Splunk4Aviation Flight Sim Lands at .conf26

If you hear someone at .conf26 shouting "gear down, GEAR DOWN" across the show floor, you have found us.  The ...

Turn Cisco Telemetry Into Action with Cisco Data Fabric, powered by the Splunk ...

The surge in machine data is already hitting enterprise budgets, and the agentic era will only intensify it. ...

Persistent Queue at TcpOut — One of Splunk's Most Practical Features

Splunk introduced persistent queueing at the tcpout layer as one of the most practical resilience features in ...