Ok, been learning alot about reducing event size from a recent conversation (here) and got linked a great article on search performance (this one) and an obvious key is reducing the events that come ...
See more...
Ok, been learning alot about reducing event size from a recent conversation (here) and got linked a great article on search performance (this one) and an obvious key is reducing the events that come back (the first line is the most important). For a lot of the reports I'll need to write, the way to do this would be the match DIRECTORY INFORMATION but that DOES NOT EXIST IN THE UNDERLYING DATA and this gets complicated with what I wrote in that other post about (2) streams of data. Here is what I mean (specifics). 1. DS 1 (call data, JSON) 2. DS 2 (policy data, JSON) 3. directory.csv (inputlookup file with data, or I could query a DB using dbxquery) So if I want to match 'mylist' in that csv then I have to do it AFTER the first line, like this: index="my_data" resourceId="enum*" ("disposition.disposition"="TERMINATED" OR "connections{}.left.facets{}.number"=*)
| stats values(sourcenumber) as sourcenumber values(disposition) as disposition by guid
| lookup directory_listings.csv number AS sourcenumber OUTPUT lists
| search lists="mylist" This brings back the (2) Datasources (the first line), but then I have to read through 100% of it, then match the directory, then filter so this is huge 'false positive' (event to scan count ratio) I've read before about using subsearch and this works great, but then leaves out one of the data sources. In other words this: index="policyguru_data" resourceId="enum*" ("disposition.disposition"="TERMINATED" OR sourcenumber=*)
[ | inputlookup pg_directory_listings.csv
| search lists="*mylist*"
| fields number
| rename number as sourcenumber
| format
]
| table * runs fast and is 1:1 event-to-scan, BUT OMITS disposition entirely, because it's not 'joining' data, but sending the sourcenumber up to the first line, which then EXCLUDES disposition because it doesn't match. Does that make sense? I suppose I could use this entire search AS a subsearch to get back 'guid' values and then pass that UP into another search but feels very...INCEPTION at that point! Anyway, looking for ideas. Thank you!