Splunk Search

Alternatives for subsearch needed (10k limit)

PPape
Contributor

Hello,

I have a powershell Script that runs every day through my Filesystem and logs every Folder with all NTFS permissions. The output of the Script looks like this: example on pastebin

Now I want to make a search with two dates and show the delta (new Folders) with all NTFS permissions.

index=filesec  |rex field=Path mode=sed "s/\\\/\//g" | table host, Path| search host="*" WER="*" (Path!="''" AND Path="*") NOT [search index=filesec earliest=-1d@d latest=@d  host="*" WER="*" (Path!="''" AND Path="*") | fields host, Path | dedup host, Path | rex field=Path mode=sed "s/\\\/\//g" | table host, Path]  | stats list(WER) as WER list(Rights) as Rights list(TYPE) as Type by Path

This one works but only if there are less than 10000 Folders in the subsearch due to the default limit of 10000 (max 10500... splunk devs have a strange kind of humor i guess). I have about 15k Folders. Any suggestions how I can make it work?

Every hint is appreciated!

0 Karma
1 Solution

javiergn
SplunkTrust
SplunkTrust

Hi,

I've done something similar in the past to calculate effective permission deltas in SharePoint, Windows, SQL Server, etc.
Try this:

index=filesec host="*" WER="*" Path!="''" Path="*" earliest=-1d@d latest=@d
| fields _time, host, Path, TYPE, WER, Rights
| rex field=Path mode=sed "s/\\\/\//g" 
| bucket _time span=1d
| eventstats dc(_time) as NumberOfDays, max(_time) as latest, min(_time) as earliest
| stats count(_time) as count, max(_time) as time by host, Path, TYPE, WER, Rights, NumberOfDays, latest, earliest
| where count < NumberOfDays
| eval ChangeType = case (
   time > earliest, "Add",
   1 == 1, "Remove"
)

Please keep in mind this is going to work fine when comparing two days.
I haven't tested it to compare more than 2 consecutive days.

View solution in original post

javiergn
SplunkTrust
SplunkTrust

Hi,

I've done something similar in the past to calculate effective permission deltas in SharePoint, Windows, SQL Server, etc.
Try this:

index=filesec host="*" WER="*" Path!="''" Path="*" earliest=-1d@d latest=@d
| fields _time, host, Path, TYPE, WER, Rights
| rex field=Path mode=sed "s/\\\/\//g" 
| bucket _time span=1d
| eventstats dc(_time) as NumberOfDays, max(_time) as latest, min(_time) as earliest
| stats count(_time) as count, max(_time) as time by host, Path, TYPE, WER, Rights, NumberOfDays, latest, earliest
| where count < NumberOfDays
| eval ChangeType = case (
   time > earliest, "Add",
   1 == 1, "Remove"
)

Please keep in mind this is going to work fine when comparing two days.
I haven't tested it to compare more than 2 consecutive days.

View solution in original post

PPape
Contributor

This is it! It works so good even with more days! Now I have to modify it to see changes in the Rights. But this helps me a lot thank you!

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

Try including format command at the end of your sub search

 index=filesec  |rex field=Path mode=sed "s/\\\/\//g" | table host, Path| search host="*" WER="*" (Path!="''" AND Path="*") NOT [search index=filesec earliest=-1d@d latest=@d  host="*" WER="*" (Path!="''" AND Path="*") | fields host, Path | dedup host, Path | rex field=Path mode=sed "s/\\\/\//g" | table host, Path|format]  | stats list(WER) as WER list(Rights) as Rights list(TYPE) as Type by Path
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!