I am running into trouble while trying to accumulate data into a csv. Things ran great for a long time, but now records are no longer accumulating properly. I am trying to accumulate data on external IP addresses scanning my company's web sites. The query ran great for months, but suddenly a bunch of data disappeared from the CSV where I had been storing it. I tried rebuilding the data, but I can't make the total amount any larger. Total query size is not large compared to other queries that I have run, but the CSV is up to 50K records, my largest CSV so far. Is there some limit to what I can do with the CSV? This is the query that I am running (with a few identifying markers removed) index=web_servers (file=*.cfm OR file=*.jsp) NOT [|inputlookup AuthorizedScanners.csv | fields + ip_address |rename ip_address as clientip] | bin _time span=10s | stats earliest(_time) as firstseen by clientip | eval reason="request for unreasonable web resources" | eval days=strftime(firstseen,"%F") | rename clientip as src_ip | append [|inputlookup web_scanning_found.csv | makemv delim=" " days |mvexpand days] | eval lastseen=coalesce(lastseen,firstseen) | sort 0 +src_ip, firstseen, lastseen | stats values(reason) as reason min(firstseen) as firstseen max(lastseen) as lastseen dc(days) as days_seen values(days) as days by src_ip | eval first_seen=strftime(firstseen,"%F %T") | eval last_seen=strftime(lastseen, "%F %T") | table src_ip reason, firstseen, lastseen, first_seen, last_seen, days_seen, days | outputlookup web_scanning_found.csv The query gives me web_scanning_found.csv, which contains - IP address of the scanner - A reason for identifying the scanner - timestamp when the scanner first showed up - timestamp when the scanner was last seen - formatted dates for the above two - number of days that the scanner has been seen in our environment - mv field of days that the scanner has been seen in our environment running this query with dates of 7/1-7/15 gives me results with 49667 records if I then run it for 7/16-7/26, the results drop to 48618 records I can rerun 7/1-7/15 and get 49667 records again. since I am just appending and doing stats by src_ip, I should always see at least as many src_ip total events should never drop, so I must be exceeding some limit. What I have tried - early on I ran into a similar problem because of the 10K limit on sort, so I added 0 to remove that limit - To figure out if I was limited in the size of a CSV, I tried removing the final outputlookup command, but still lost records, so the limit is not in what the csv can hold Anyone have an idea where I am going wrong? What else can I try? ... View more