Bin limits?

[mistydennis]

I'm running a search over two sourcetypes - an inputlookup (contains metadata about the pdf) and the log file (contains download counts of the pdf) in order to show what pdfs are being downloaded over a given period of time. I've applied a bin function to the download counts so that any distinct IP is counted once during a 30 minute time period (to eliminate possible bots). I'm finding that the bin only works if the time picker is set to less than 4 months. As soon as I run a search for download counts OVER a 4 month time period, only about 70% of the hits are counted. Here's what I'm using:

| inputlookup 'foo' | join type=left 'pdfName' [search response_code=200 | bin _time span=30m | stats count dc(src_ip) as distinctip, dc(pdfName) as DownloadCounts by pdfName, agentType, IPLocation, _time, src_ip | stats count AS Downloads by pdfName] |table pdfName, pdfTitle, pdfAuthor, Downloads 

What am I doing wrong? Is there a bin limit? Or is this a complication of using join? Any help would be greatly appreciated.

[somesoni2 · Accepted solution]

Try this. For best performance, it's often recommended that you add one or more metadata field filters in the base search, such as index, sourcetype, source or host. So, if possible add those to your log file query.

index=foo sourcetype=bar response_code=200 | bin _time span=30m | stats count by pdfName, agentType, IPLocation, _time, src_ip | stats count AS Downloads by pdfName | append [| inputlookup 'foo' ] | stats values(*) as * by PdfName

[mistydennis]

This didn't give me exactly what I wanted but it pointed me in the right direction. I ended up using:

index=foo response_code=200 | bin _time span=30m | stats count by pdfName, agentType, IPLocation, _time, src_ip | stats count AS Downloads by pdfName | join [|inputlookup 'foo' |]

The append command gave me two rows (one was empty) but the join command gave me a single line.

I really appreciate the help, thank you so much!