Dashboards & Visualizations

How to label search results and create a "Time since last crawls" dashboard panel?

pentago
New Member

2 questions

1. I search my Nginx access logs for various Googlebot traffic (web, image, ads, mobile, smartphone, adsense, feedfetcher).

All of those bots have their own (https://goo.gl/sUcGvU) useragent which I use to perform the search. Currently I just paste those useragents and get results, but I'd like to get search results labeled.

For instance, all search results for "Googlebot/2.1 (+http://www.google.com/bot.html)" I'd like to be labeled as "Googlebot" in statistics tab.

Example:
alt text
alt text

2. Is there a chance to create somehow dashboard panel like this or smililar? alt text

0 Karma

Runals
Motivator

I'd use a case statement and a regex match or maybe like against whatever your user agent field is called

sourcetype = nginx | eval bot = case(match(useragent,"Googlebot"), Googlebot, match(useragent, "foo"), "Foobot", match(useragent, "Bar"), "Barbot", 1=1, "Unknown Bot")

I'd have to think about your dashboard but I'd probably start with something like

sourcetype = nginx | eval bot = case(match(useragent,"Googlebot"), Googlebot, match(useragent, "foo"), "Foobot", match(useragent, "Bar"), "Barbot", 1=1, "Unknown Bot")  | stats max(_time) as last_seen by bot | eval current_time = now() | eval delta = current_time - last_seen | timechart max(delta) by bot 

The problem is that search doesn't account for the number of days ago by individual days so you might need to bake in a bins command before the stats and actually convert the stats to eventstats. I'm not sure what the colors mean; I'm guessing individual bots.

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust
  1. Use eventtypes. Once you define an eventtype, use it like such: stats count by eventtype | where eventtype="bot_*"

    [bot_googlebot]
    search = Googlebot/2.1

http://docs.splunk.com/Documentation/Splunk/6.2.3/Admin/Eventtypesconf?r=splunky

  1. Yes. But for performance reasons, you would probably want to use a Data model acceleration or Summary Indexing to do the relevant counts. Summarize the data into daily buckets. Also, it depends on what your time ranges are refering to. Is it counts of activity? or ...?
0 Karma

pentago
New Member

Awesome. Thanks man. Any chance i could rename those searches directly from search? I saw that some stuff could be renamed with AS or something like that. Not quite sure as I'm a newbie.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...