Splunk Search

Why am I getting different results for the same search

eckdale
Path Finder

If I run the same search using two different time windows I consistently get different results.
I'm looking to count the number of email messages sent. The search query is simple:

index="linux" sourcetype="syslog" | transaction queue_id

When i search for a time range of 19:00:00 to 19:30:00 I get the following results (note from 19:21 on there are zero results):
alt text

If I then search for 19:21:00 to 19:22:00 I get the following results (note: now where are 245 events):
alt text

0 Karma
1 Solution

woodcock
Esteemed Legend

Because transaction is a command that consumes a HUGE amount of RAM and when scaled to any practical time-window will exhaust all RAM available to you and your search and WILL SILENTLY ABORT yielding partial results and NO INDICATION THAT HAPPENED (unless you Inspect your job and check the search.log). This is the main reason that I am constantly harping: DO NOT USE transaction! So don't use it.

View solution in original post

0 Karma

twinspop
Influencer

Adding to what Woodcock said above, I've found transaction can be replaced by stats in most situations with a little extra effort. Obviously it depends on what you're trying to report on, but something like this:

index=linux sourcetype=syslog 
  | stats last(somefield) as last_something first(somefield) as first_something range(_time) as span by queue_id
0 Karma

woodcock
Esteemed Legend

Because transaction is a command that consumes a HUGE amount of RAM and when scaled to any practical time-window will exhaust all RAM available to you and your search and WILL SILENTLY ABORT yielding partial results and NO INDICATION THAT HAPPENED (unless you Inspect your job and check the search.log). This is the main reason that I am constantly harping: DO NOT USE transaction! So don't use it.

0 Karma

eckdale
Path Finder

Per your instruction I took a look at Inspector (should have done this originally) and it does indicate: "Some transactions have been discarded. To include them, add keepevicted=true.

Adding that to the query does in fact resolve the issue.

0 Karma

woodcock
Esteemed Legend

Even so, do not use transaction; take the time and effort to use *stats.

0 Karma

eckdale
Path Finder

So that's good to know. I'll inspect the job and look into search.log for more details.

Two questions then:
If I can't use transaction to group multiple events then what do I use?
My single-instance Splunk Enterprise server has 256GB of memory and appears to use very little of that memory. Am I really exhausting resources?

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...

Design, Compete, Win: Submit Your Best Splunk Dashboards for a .conf26 Pass

Hello Splunkers,  We’re excited to kick off a Splunk Dashboard contest! We know that dashboards are a primary ...

May 2026 Splunk Expert Sessions: Security & Observability

Level Up Your Operations: May 2026 Splunk Expert Sessions Whether you are refining your security posture or ...