Getting Data In

Write to summary index, Why are some fields are not populating?

tlmayes
Contributor

Have a query comprised of 2 subqueries (joins).  Output is exactly as expected
When I try to push that data to a summary index, only the fields from the original query make it, for all fields and event data generated from the sub queries there is nothing.    Finally, when I run the query (including '|collect index=summary' as the last line) everything expected is in the output, just not making it to the summary index.

 

 

 

index=blah_blah <followed by a search>
| join [<search string1> [ <search string 2]]
| fields _time IP DNS NETBIOS TRACKING_METHOD OS TAGS QID TITLE TYPE SEVERITY STATUS LAST_SCAN_DATETIME LAST_FOUND_DATETIME LAST_FIXED_DATETIME PUBLISHED_DATETIME THREAT_INTEL_VALUES THREAT_INTEL_IDS CVSS_V3_BASE VENDOR_REFERENCE RESULTS 
| collect index=summary

 

 

 

Output is fully populated, yet summary index is missing several fields (and the associated data).

Note: the missing fields in the summary index are all from the sub-searches/join.

 

0 Karma

tlmayes
Contributor

Interesting, and thanks for the reminder (I forget about the job inspector).  No smoking gun, other than the fact it says is wrote "1,000" results??  The query returns 60,000 events.  Is there a limit to how much you can write to a summary?

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Which query? There is a limit of 50,000 events in a subsearch - could that be your issue?

0 Karma

tlmayes
Contributor

You actually provided a solution several weeks ago that resolved the query count problem for subsearches.  Sub-search#1 produces ~ 1000 events.  Outcome of sub-search#2 produces ~ 4,500 events.  The final search produces ~60,000 events (the same query that ends with "|collect index=summary"

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Are there any indications in the job inspector as to what may have happened?

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Laser Bananas and Edge Hubs: Exploring Operational Technology (OT) Data Through a ...

  OT is a different environment to traditional IT and can have interesting challenges when interfacing the ...

Event Series: Mastering AI Tokenomics and Splunk Agent Observability

Beyond the Black Box: Correlating AI Performance and Tokenomics with Splunk Agent Observability   As ...

span_metrics: The OpenTelemetry-Idiomatic Way to See Inside Your Services

You open a trace in Splunk Observability Cloud and everything looks fine. One root span, order-pipeline, with ...