Security

Search ends before it gets to the data cap (500MB)

CarsonZa
Contributor

I have a user that is attempting to run a very large search. I raised the role srchDiskQuota but the search is still ending before it gets anywhere near the data cap. The role limit is set to 500MB and the search stops at 243.66MB. I've ensured that the user doesn't have any saved / back grounded jobs as well. I've tried the search on my account and it does the same. Any thoughts on what might be causing this?

index=iis c_ip!="10." c_ip!="192." session_nonce!="" cs_uri_stem="*.php" cs_uri_stem!="*piwik.php" cs_uri_stem!="*ee-atlogofetch.php"
| eval Platform=if(substr(cs_User_Agent, len(cs_User_Agent)-13)="Mobile-Wrapper","Mobile","Online")
| eval session_nonce=substr(session_nonce, 1, 32)."-".c_ip
| eval client=substr(client, 1, 5)
| stats first(client) as client, first(user) as user, first(Platform) as Platform, first(date) as date, max(_time) as MaxTime, min(_time) as MinTime by session_nonce
| lookup Employee_Lookup user as "user" OUTPUT hr_eename as ee_name, hr_eestatus as ee_status, LOS as LOS, hr_deptdesc as dept_desc, hr_position as position
| eval hr_eename=if(len(hr_eename) > 2, hr_eename, user)
| search client=08001
| eval session_duration=tostring((MaxTime-MinTime),"duration")
| convert ctime(MaxTime), ctime(MinTime)

0 Karma

Vijeta
Influencer

Check your limits.conf for data size

0 Karma

CarsonZa
Contributor

What stanza am I looking at?

0 Karma

pradeepkumarg
Influencer

How exactly do you know that the search has stopped?
What does the job inspector say? There are numerous other limits enforced like maxSearchTime and number of records you can pass in a join or subsearch.
The job inspector should clearly say what limit was enforced for that particular run.

0 Karma

CarsonZa
Contributor

I know it stops because i get an error that says "search auto-finalized after disk usage limit (500MB) was reached." but the size of the search only gets to ~243MB. A time limit enforcement is very plausible because of the amount of events that are being returned but i wouldn't expect to get an disk usage message.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...