I'm having really hard time figuring out how this web intelligence app should be installed.
I have my logs indexed in the "default" index of Splunk. Installed web intelligence. All shows empty. Checked source type, and I have plenty of logs on that.
Ran backfill script, it finished. But web intelligence is still empty. Checked indexes, and all 3 that belong to web intelligence show 0 rows.
I don't understand where would the data in those indexes come from? Did I not set up something correctly? Where is the manual on all of this?
After setting up log forwarders from production servers, all reports started working when time period is set to "Real time." When changing to "Today" or some other timeframe, it does not work. Indexes started growing too, so data is trickling in. But something's not right for some reason.
Could it be related to the bad log format? How do I view what is "access_combined" definition in Splunk so that I can check against my actual files?
I had problems initially because my apache logs were not in the default format. Are your logs being properly sourcetyped as access_combined? Also - did you run the setup portion of the app - where you indicate your indexes and sourcetypes?
I had the same situation originally - I could see my logs in the preview but none of the charts were populating. In my case it was because some of the default fields were not being parsed which caused the searches to fail to match. Did you try running one of the default searches manually under Data Exploration -> Search to see if results were returned?
All of them have index reference in them. All of Web Intelligence's indexes are empty. That's what I don't understand... how should they get populated, and where to check why this is not happening?
Just running this, for example: 'source="Web Traffic*"' returns 0 results...
The webintelligence indexes are all summary indexes. The realtime searches operate on your default index - where ever your access_combined logs are. The summary indexes only get populated by the scheduled regenerator searches - if these searches fail to retrieve results then the indexes will not be populated.
Real time dashboard has this in query: eval search=if(range<=(86400+3600),"index=wisummaryhourly","index=wisummarydaily")
I don't know splunk search language, but it clearly looks like it would use either one index or another.
The realtime dashboard has a time picker which sets which search to use. wisummaryhourly, wisummarydaily and wisummaryfivemin are the summary indexes and all get populated by the scheduled regenerator* searches - there are about 40 of them.
Check your $splunk_home/var/spool/splunk directory - this is where the cached files (stash files) are written prior to being written to the summary indexes. If there's no files in this directory than the searches aren't working.