How to set up a summary index?

[soundchaos]

I read all the splunk documentation for setting up a summary index, and I followed it as best I could, but I cant get results when I try to search against it.

My search: index="summary" search_name="404_logs"

but my search is not even listed in any indexes with index="summary*"

If I go to settings>knowledge>searches, reports, and alerts,
It shows my 404_logs search that I am trying to set up as a summary index, and it has 0 alerts.
(it has been over 24 hours since I set it up)

In that search, it is configured as follows:

SEARCH: index="is_logs" source="mysite.com" sc_status = 404
DESCRIPTION: Summary Index of 404 errors
Not accelerated
SCHEDULE: -1y to now, basic, every day at midnight.
ALERT: Condition - Always, alert mode - once per search, no throttling, 24 hour expiration, medium severity
ALERT ACTIONS: All disabled
SUMMARY INDEXING: Enabled, index - summary, add fields - blank

I'm not sure if I am trying to search against it improperly, or if it is not set up right. edit: My eventual goal is to be able to easily pull up a time chart of 404 errors within the last year, because without using summary indexing, the search takes over an hour to complete on the dashboard every time the page is loaded, and I need to use the 404 error data in other searches as well.

[soundchaos]

I don't have access at all, but I'm working with my system admin to get this. I'm not sure he will be able to find it, have any instruction I could give him?

[somesoni2 · Accepted solution]

I see following possible issue with your summary index search configuration (not necessarily for the issue that you're facing):

1) The SEARCH is not summarizing anything. You should use the some aggregate command to summarize data so that later when you use index=summary it has to retrieve/process less data.

2) The time range for search should be according to schedule. e.g. for daily schedule, it should select last 1 day data, else you will have duplicates.

My suggestion would be (based on the requirement that you need the summary for timechart).

SEARCH: index="is_logs" source="mysite.com" sc_status = 404 | timechart span=1h count
DESCRIPTION: Summary Index of 404 errors
Not accelerated
TIMERAGNE: -1d@d to @d, 
SCHEDULE: basic, every day at midnight.
ALERT: Condition - Always, alert mode - once per search, no throttling, 24 hour expiration, medium severity
ALERT ACTIONS: All disabled
SUMMARY INDEXING: Enabled, index - summary, add fields - blank

To get data for last year, you should backfill the summary index.
http://docs.splunk.com/Documentation/Splunk/6.1.3/Knowledge/Managesummaryindexgapsandoverlaps

[soundchaos]

08/19/2014 06:00:00 -0600, search_name=404_logs, search_now=1408514400.000, info_min_time=1408428000.000, info_max_time=1408514400.000, info_search_time=1408514567.910, count=836

and

2014-08-19 05:59:44 W3SVC2....|utmccn=(direct)|utmcmd=(none);+RequestVerificationToken_Lw=;+ASP.NET_SessionId=...;+RSA=...;+.RequestVerificationToken=... - ... 404 0 2 1397 1014 249

(... replacing all the numbers, user agent, and keys that were too long to paste here)

[somesoni2]

can you post some raw events that you get from by executing 'index=summary source="404_logs"' ? (results from summary index search)

[soundchaos]

Yes, the original summary index search produces a good timechart, and the stats view of it does show the count. Its just when I reference the original through index=summary source="404_logs", I just get a normal list of raw results and no count field. Also, no sc_status field, so I cannot rebuild a timechart with the results either. I also misread the results when I said making progress, I do NOT get a _time field with a 1hr time span, as far as I can tell now

[strive]

does your search used for summary indexing produces results? I think that's the first place to start your troubleshooting. Take the search and run it in search window to see if it is producing any output.

[soundchaos]

Making progress! I do get the 1hr span time field now, but no count field, so that timechart isn't working

[somesoni2]

Once you have setup this (and backfill as required), the index=summary source="404_logs" should give you following fields : _time (1hr span) and count. To get timechart of this data, you can do this.

index=summary source="404_logs" | timechart span=yourTimeSpan sum(count) as count

[soundchaos]

That gets me a good place to start, and I should be able to do the backfill with no issues.

But now that I have set the time range to 1 day, I still cant find a way to search against this data.

[strive]

Do you have access to your savedsearches.conf file? Could you post that settings here. Post complete stanza.