Index vs Search Time for Busy Web Logs

[Kyle_Brandt]

I am moving my web log reporting to Splunk. Even when I don't log static content I have about 1.5 Million events per hour with about 20 fields per entry I want to extract.

In the interest of speeding this up I have made one search time regex which contains all the fields instead of multiple regexes.

In the end I would like to be able to run multiple report like "eventtype="HAProxy Web Logs" | chart count by HTTP_CLIENT_IP | sort count desc" over 24 hour periods. However, even running one of these reports for the period of 1 hour takes several minutes.

In order to optimize searches and reporting on this amount of data am I going to have to create index time field extraction? Are there other optimizations I should be looking into?

[gkanapathy · Accepted solution]

Index time field extractions will not help and in fact will probably make things slower. To be able to efficiently report, you will need to use summary indexing. Summary indexing is not too hard to set up, but it does cost you flexibility as you essentially decide what sort of things you'd report on first, then build the summaries.