Splunk Search

Bug? splunk advanced searching/views does not display correctly

Communicator

Hi...
Its been a while I have problems with searching in Google maps or geoip which the thread was going on here: geoip search results not correct

if u follow up the thread, u see it came to a point that we all realized there exists some sort of limit that does not let geoip or Google maps to display more than ten thousand...
today suddenly, I realized its not only geoip/Google maps, but it actually is any kind of advanced searches. for instance if you search for all the data in normal search using "*", and if you have huge number of indexed data, (i.e. billions of data), u'll probably see all in the search but if you change the view to "Advanced Charting View" then you'll only see partially few thousands of those results...
Im not sure whether this is a bug or if there is some sort of limitation in any file... but that definitely causes major problems.
Does anyone have any idea how to solve this issue..?

Tags (2)

Communicator

thanks hexx for detailed information and references you provide here.. but this is getting even weirder... 😄

although the command is used by all the users without the lookup command, I tried your way and received error:

[EventsViewer module] Error in 'lookup' command: The lookup table 'geoip' does not exist.

and trying the same command in Google Maps gives this error:

Rendering...
Error : Traceback: 
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/maps/appserver/modules/GoogleMaps/GoogleMaps.py", line 53, in generateResults
    for result in getattr(job, entity_name)[offset:end]:
  File "/opt/splunk/lib/python2.6/site-packages/splunk/search/__init__.py", line 1280, in __getitem__
    self.job.pushValidation()
  File "/opt/splunk/lib/python2.6/site-packages/splunk/search/__init__.py", line 610, in pushValidation
    raise splunk.SearchException, fatality
SearchException: Error in 'lookup' command: The lookup table 'geoip' does not exist.

besides, the SPP about page, located at .../app/maps/about is a help document with this search as example:

Perform a geolocation lookup for
values of the clientip field in
accesscombined events:
sourcetype=access
combined | geoip
clientip

Same as the previous example, but also
perform DNS lookups in case when the
value of the clientip field is a
hostname and not an IP:
sourcetype=accesscombined | geoip
clientip resolve
hostnames=true

Same as the first example, but using
the geo lookup instead of the command
sourcetype=access_combined | lookup
geo ip as clientip

etc... etc... etc...

I even tried this:

SIP="*" | lookup geo SIP

and got the same error...

0 Karma

Communicator

yup dmaislin, and yet no difference... although I knew problem is not from Google Maps permissions as the geoip command behaves the same both in Google Maps and the flashtimeline search.
thats why I pointed my problem to geoip, not google maps..
the snapshots that i provided here on geoip as well was done on the flashtimeline, not Google Maps...

0 Karma

Splunk Employee
Splunk Employee

Are you in the google maps app when you run this command? Or, did you change the permissions on the App and make it Global yet?

0 Karma

Communicator

Alright... since this problem behaves exactly the same with any kind of searching I do, I start from a very simple search first...

I am applying field extraction using DELIMS, hence I have a field called SIP which stands for sourceip...

so now, what I want to show u is results for SIP=* for a normal search, then results for same search on advanced charting view, then SIP=* | geoip SIP to also have geoip behaviour...
as you can see, both geoip and advanced chart only retrieve 10000 results !

these are snapshots for the normal search where you can see is up to 7 millions and I had to actually stop it since there was already enough results and it was taking much time... but the point is, the data that exists is way more than 10000...

alt text
alt text
alt text
alt text

then here it is on advanced charting:

alt text
alt text
alt text
alt text

and last but not least, geoip!

alt text
alt text
alt text
alt text

as you can see, both advanced charting and geoip have only 10000 results! the matching event is different. which indicates 2 problems actually:

  1. both of them stop only on counting 10000 results
  2. accuracy of these 2 as well is now a question mark! because if they both only fetch 10000 results, given that my search in both was the same, their matching events as well should be exactly the same!!
0 Karma

Splunk Employee
Splunk Employee

Looking at the MaxMind geoip app, I'm not sure that you are invoking the geoip command correctly. As far as I can tell, geoip.py is an external command used by the geoip lookup, not intended to be a search command. It should therefore be invoked with the lookup command. What if instead of :

... | geoip SourceIP

...you run :

... | lookup geoip SourceIP

0 Karma

Communicator

thank you for the update.
if the user wishes to maximize that for any reasons of their own, how would that go..?
besides, would you explain as well geoip's behavior as geoip command entered in flashtimeline also stops after 10000 results...

0 Karma

Splunk Employee
Splunk Employee

Thank you, now I know what's going on. Please review my amended answer.

Splunk Employee
Splunk Employee

The display of result rows will be limited on a per-command basis and is typically configurable in limits.conf with maxresultsrows for stanzas such as [searchresults] or [stats]. Now, even if the display of results rows is truncated, reporting commands such as stats will still show accurate aggregates, which take all input events/results into account.

For more information and details, I would recommend to read this Splunk Answer as well as this one.

Update: Adding the information below to clarify the purpose of the advanced charting view.

The important thing to understand is that the advanced charting view is designed to process the results (not the raw events!) produced by a reporting command. This is why you will see the exact same search which would show millions of events in the flashtimeline return an approximate maximum of 10,000 results in the advanced charting view. Now, if you take that same search and pipe it to | stats count, you will see that the search powering the advanced charting view will indeed process all expected events into results before letting you decide how to render those into a chart.

To sum it up : Do not use the advanced charting view to render and view events, it is not its purpose. Instead, use it to experiment with different visualization methods to apply on the results of your reporting search.

0 Karma

Communicator

Draineh,

here the limits.conf results in pastebin

Hexx,

I'll get that and post it here ASAP. thanks

0 Karma

Splunk Employee
Splunk Employee

@nina15 : Just to be sure, could you tell us exactly what search string you are feeding into the advanced charting view? Could you tell us how many events the search reports to have found? Ideally, I'd like to see a screenshot of the search job inspector output.

Champion

Could you run the command
./splunk cmd btool limits list --debug
and pastebin the results with a link here please?

Communicator

thanks for your response hexx,

as I'd stated in my description the view causes the problem... which is in line with your say that using other search commands (stats, timechart, etc) gives more results... I explained in my other post (the link is available above), the problem seems to raise when it has to display more than that number of results, not counts and stats... (I've already explained this in detail in my other post)

and again, in other post I did mention that I tried all possible parameters in limits.conf which includes the ones you are saying, and yet it did not have any effects neither on advanced charts nor geoip!!!

0 Karma

Communicator

besides, if thats the case, why geoip behaves the same way... its very unlikely its a coincidence...

0 Karma

Communicator

I know I tried many sorts of searches that should have shown thousands of results... but on using stats or timechart I have to try first and get back to u... but if there is any limits anyways, wouldnt it prevent from all kinds of searches...?

0 Karma

Splunk Employee
Splunk Employee

What search are you running that seems to limit its output to 10,000 rows in the advanced charting view? I am fairly certain that this limit is imposed by the default rendering of the advanced charting view, but if you were to run your search through a reporting command such as stats or timechart, you would get your full set of rows.

Champion

Looking at the module references the is a limit within the XML that you can define although you want to be careful how you adjust this as it can have an impact on the performance. Hopefully with 4.3 this won't be as large an impact thanks to the HTML5 visualisations but I expect the calculations in the background also have an equally large impact;

            <module name="FlashChart">
                <param name="width">100%</param>
                <param name="height">400px</param>
                <param name="maxResultCount">10000</param>
            </module>

Also here, http://splunk-base.splunk.com/answers/10349/chart-only-showing-1000-events , Nick has some ideas on how to expand the number of results in other ways (Nick is pretty much the expert on all things XML related)

Communicator

thanks so much dmaislin for responding... i really was feeling im loosing it...
anyways, i've already opened a support ticket. the number is CASE [73624].
thanks alot for ur support 🙂

0 Karma

Splunk Employee
Splunk Employee

Nina,

Can you open a support ticket? Let me know the ticket is and I will escalate for you.

Communicator

still no answers/opinions...???

0 Karma

Communicator

from what I understand, that post is only about charting, time ranges and XML...
my problem I think resides in lower level of splunk, is not the matter of how it is represented, but the problem is that the results are not fetched at all... the poster of that thread had a problem that the results were shown in the table, but not on the chart, only.
mine is not displayed anywhere when I search in Advanced charting or when I use geoip in the normal search...

0 Karma