Alternatives for subsearch needed (10k limit)

[PPape]

Hello,

I have a powershell Script that runs every day through my Filesystem and logs every Folder with all NTFS permissions. The output of the Script looks like this: example on pastebin

Now I want to make a search with two dates and show the delta (new Folders) with all NTFS permissions.

index=filesec  |rex field=Path mode=sed "s/\\\/\//g" | table host, Path| search host="*" WER="*" (Path!="''" AND Path="*") NOT [search index=filesec earliest=-1d@d latest=@d  host="*" WER="*" (Path!="''" AND Path="*") | fields host, Path | dedup host, Path | rex field=Path mode=sed "s/\\\/\//g" | table host, Path]  | stats list(WER) as WER list(Rights) as Rights list(TYPE) as Type by Path

This one works but only if there are less than 10000 Folders in the subsearch due to the default limit of 10000 (max 10500... splunk devs have a strange kind of humor i guess). I have about 15k Folders. Any suggestions how I can make it work?

Every hint is appreciated!

[javiergn · Accepted solution]

Hi,

I've done something similar in the past to calculate effective permission deltas in SharePoint, Windows, SQL Server, etc.
Try this:

index=filesec host="*" WER="*" Path!="''" Path="*" earliest=-1d@d latest=@d
| fields _time, host, Path, TYPE, WER, Rights
| rex field=Path mode=sed "s/\\\/\//g" 
| bucket _time span=1d
| eventstats dc(_time) as NumberOfDays, max(_time) as latest, min(_time) as earliest
| stats count(_time) as count, max(_time) as time by host, Path, TYPE, WER, Rights, NumberOfDays, latest, earliest
| where count < NumberOfDays
| eval ChangeType = case (
   time > earliest, "Add",
   1 == 1, "Remove"
)

Please keep in mind this is going to work fine when comparing two days.
I haven't tested it to compare more than 2 consecutive days.

[PPape]

This is it! It works so good even with more days! Now I have to modify it to see changes in the Rights. But this helps me a lot thank you!

[renjith_nair]

Try including format command at the end of your sub search

 index=filesec  |rex field=Path mode=sed "s/\\\/\//g" | table host, Path| search host="*" WER="*" (Path!="''" AND Path="*") NOT [search index=filesec earliest=-1d@d latest=@d  host="*" WER="*" (Path!="''" AND Path="*") | fields host, Path | dedup host, Path | rex field=Path mode=sed "s/\\\/\//g" | table host, Path|format]  | stats list(WER) as WER list(Rights) as Rights list(TYPE) as Type by Path