All Apps and Add-ons

No results found using Web Intelligence app

yumology
Path Finder

I have installed the Web Intelligence Beta app and I am unable to see any data within it. In fact I see "no results found" on every page and view.

I have completed the setup as instructed. I have my apache secure_access.log and access.log files being monitored and sent to splunk. I can see both of these sources from within splunk. These are categorized in the access_combined sourcetype. I even see the logs in the preview during the setup of this app. I have tried generating a lot of web traffic after the setup was complete and even backfilling the data but haven't spotted a single piece of data from within the app yet (unless I do a custom search).
I have now spent 5 days on this and about 10 hours trying multiple combinations of configurations and I'm now pretty frustrated with this. Can someone please help me identify where my problem lies? Thanks.

pde7
Explorer

Check your Apache logging format (httpd.conf). The jobs running behind the tables require the "combined" format. You may be in "common" format.

The jobs are using search filters based on referrer or client UI. This causes an empty result set if your logs are in "common" format.

A simple way to test this is to try comparing the following searches in the web intelligence search window: "eventtype=pageview eventtype=ua-browser-*" vs. "eventtype=pageview". If you have no results on the first one but plenty of results for the second one, then the jobs I'm talking about are likely failing with no results.

0 Karma

oscarspaz
Explorer

I spent almost a week trying to figure out how to get Web Intelligence to display data from apache log. Today, I finally be able to get some results in pageview. I just want to share what I have found and hopefully it may help you.

All the problems so far come from missing eventtype or assigning wrong eventtype. Need to edit the /default/eventtypes.conf under the webintelligence application folder to make the app work. I tried using a /local/eventypes.conf file but for whatever reason it will not override the default/eventtypes.conf as it should.

a. A number of the search queries in the app depend on eventtype=web-traffic. The default set up for the stanza [web-traffic] is

[web-traffic]
search = sourcetype="My Access Logs"

That means only logs that has sourcetype="My Access Logs" will be assigned web-traffic eventtype!
Therefore if this stanza is not changed to (for apache log)

[web-traffic]
search = sourcetype=access_c*

you will see no results in Pageview and other predefined searches in the app that depends on web-traffic eventtype.

b. If you are testing the app on your own internal network like I was doing, need to redefine stanza [client-nonroutable], [clientip-internal] to not matching your local IP address. This is because search queries like Pageview excludes any events that are assigned with eventtype=[client-nonroutable] or eventtype=[web-traffic].

c. Check the browser section in eventtype.conf to see if any of the user agents match the user agent part in apache log. If none of them matches, add a stanza for your browser. I found that some search queries depends on eventtype=[ua-browser-*]. If it is missing, no search results.

The test apache log I used have user agents from Chrome/16.x.x and from Firefox/8.x and they fail to match a specific browser stanza.

d. Need to restart splunk everytime for any changes in the eventtypes.conf file.

The set up document needs to be updated to include editing the eventtype.conf file. It will not work out of the box by just following the set up instructions.

This is my experience. Good luck!

sebastian_stute
Engager

it isn`t. We have openSuse here installed and used the Sampledata.zip from the Splunk Tutorial which even did not work.

@Splunk Team: please fix or tell us how to succeed!

oscarspaz
Explorer

I did followed the Set up work flow and get no data in the app just like the first post by yomology. I digged deep into the app and these are the things that I found.

I am beginning to think this may be a Windows only problem.

araitz
Splunk Employee
Splunk Employee

It seems like you didn't try to use the app's own Setup. Is there a reason for that?

0 Karma

stenorman2001
Engager

I believe I've found the problem, at least in my case. The problem is that our Apache log isn't recording the referrer or user-agent in the log file. The searchers used by the app seem to require this information, including the backfill ones which execute a bunch of saved searches.

I was able to get around the user-agent one by creating a new Event Type in Manager -> Event types called ua-browser-none with a search of of "*" (just the star). This causes the searches to accept anything in that field (or nothing). While I haven't tried it yet, I'm guessing the referrer might be solvable in a similar way if it is blocking other searches from being performed.

Could someone from Splunk provide the log format for Apache that this app is expecting (such as how the LogFormat command would appear in the appropriate Apache .conf file)?

0 Karma

yannK
Splunk Employee
Splunk Employee

I don't have the list of the required fields, but
here is a list of fields expected in the app

(we can see how the IIS fields are renamed to match)

[(?:::){0}iis-*]

FIELDALIAS-status = sc_status AS status

FIELDALIAS-clientip = c_ip AS clientip

FIELDALIAS-host = cs_host AS host

FIELDALIAS-uri = cs_uri_stem AS uri

FIELDALIAS-uripath = cs_uri_stem AS uri_path

FIELDALIAS-q = cs_uri_query AS q

FIELDALIAS-referer = cs_Referer_ AS referer

FIELDALIAS-refererdomain = cs_Referer_ AS referer_domain

FIELDALIAS-useragent = cs_User_Agent_ AS useragent

FIELDALIAS-method = cs_method AS method

0 Karma

stenorman2001
Engager

I am also having the same problem with this app. After installing the app and backfilling the data (which takes ages!), the app does not show any data, despite the fact I can see it if I do a manual search from within the app.

yannK
Splunk Employee
Splunk Employee
  • what is your version of splunk ? (I think that you need at last 4.2)

  • when you go to the search app, can you see logs when using sourcetype=access_combined in the default indexes ? (if they are in another index, make sure to add it to the default searchables index list in the account manager)

  • if nothing is loaded, please check the internal logs for errors( using index=_internal source=*splunkd.log )

    • if the summary indexing working in general (did you disabled the inputs on the splunk spool folder ?)
0 Karma

yannK
Splunk Employee
Splunk Employee

Hum, if you are using the free version (with an expired trial key), then the scheduled searched generating the summary data are not running. This is probably the cause.

yumology
Path Finder

Still unable to get this app to work. Perhaps it cannot function on the free version of splunk? Anyways, is there another web statistics app out there? If not then I'll just have to build my own app and give up on any help here.

0 Karma

yumology
Path Finder

I have been using the admin account this whole time so it shouldn't be a permissions issue.
I originally started with a forwarder sending data to my indexer. The web intelligence app is installed on the indexer. During my troubleshooting I set up apache web server on the indexer itself and am monitoring a local apache log also. So both local monitoring and receiving logs from a remote forwarder are not seeing the summary indexes.

0 Karma

yannK
Splunk Employee
Splunk Employee

to resume:
- the schedule searched are running, generating files in the spool folder, that are indexed (then deleted)
- but when you search in the summary indexes on the server they are empty.

2 things to check :

  • are you authorized to search on those indexes (check your role permissions, or try as admin) ?

  • what is your server :an indexer, a search-head, a forwarder ?
    Please verify that it is not forwarding the events to another server (that is dropping the events because the indexes don't exits there, or is not a search-peer, therefore is not returning any results)

0 Karma

yumology
Path Finder

There has been no progress on this since. I am still looking for help on this and when I finally solve it I will post what I did to do it so there's at least some kind of documentation out there. I tried setting the environmental variables by hand by editing /etc/environment but this didn't result in anything. Any other ideas on how to solve this?

0 Karma

yumology
Path Finder

The stash_new monitor is on and enabled. There are no files in the $SPLUNK_HOME/var/spool/splunk/ directory. Also from the bash shell in Ubuntu, "echo $SPLUNK_HOME" doesn't show anything and I wonder if there's an environmental variable problem here. Another thing I found was that there is absolutely no data in any of the wi_summary_* indexes. This is probably my problem but I don't know how to fix this. Any ideas on why they aren't populating? Thanks!

0 Karma

yannK
Splunk Employee
Splunk Employee

The last question was about checking if the summary indexing is working.

to resume the process of summary indexing :

scheduled searches (with summary indexing option) -> write results in the folder $SPLUNK_HOME/var/spool/splunk/...stash_new -> this folder is monitored and the results stored in the specified summary index then deleted from the spool folder -> the index is used in the dashboards of the app.

So please check that the monitor on the folder is not disabled (manager > data inputs > file inputs)
then check that the indexes wi_summary_* exists and contains results.

0 Karma

yumology
Path Finder

The newest version of splunk 4.2.4.
Yes I see events when I search for sourcetype=access_combined and they are in the main index as expected.
There are no issues in index=_internal that I see.
I don't know what you mean about checking splunk indexing or the splunk spool folder.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...