Fire Brigade Version 2: Why is fb_hostname_index_cache blank?

[ccsfdave]

I am trying to figure out how this file gets populated. Mine is blank.

A previous answer stated that dbinspect populated it, but I cannot find data on hosts when I run that command. In the same directory, monitored_indexes is populated.

Can anyone help draw the lines for me so I can track down the population of the fb_hostname_index_cache?

Thanks.

Dave

[ppablo]

FYI, Fire Brigade version 2 will no longer be updated (latest version is 2.0.3). The newer versions 2.0.4 and higher will now be available with the original “Fire Brigade” app on Splunkbase which was just updated to support Splunk 6.3. This is noted on the page for Fire Brigade on Splunkbase:
https://splunkbase.splunk.com/app/1581/

If you have any questions, ping the developer of the app @sowings

Cheers!

[ccsfdave]

So, to boil down the conversation:

I needed to install the TA-fire_brigade on my indexer linked here: https://splunkbase.splunk.com/app/1633/

I then ran (because I don't have patience):

curl -k -u admin:pass https://:8089/servicesNS/admin/TA-fire_brigade/saved/searches/Update%20monitored%20list%20from%20RES... -d trigger_actions=1

and I ran:

curl -k -u admin:pass https://:8089/servicesNS/admin/TA-fire_brigade/saved/searches/DB%20inspection/dispatch -d trigger_actions=1

And that did it!

[sowings · Accepted solution]

From the TA (placed on indexers, or standalone search heads), the "Update monitored list from REST" saved search runs just before midnight to populate the monitored_indexes.csv. The same host (still the TA) then uses that list a few minutes later (just after midnight) to run dbinspect over each index. This data is stored in the summary index.

Later, the main Fire Brigade app on a search head referencing these indexers (or in a single, standalone instance, where both the TA and main app would be installed side-by-side), runs a search called "FB - Host Index Cache" to populate that lookup. This is used for dropdown menus and the like.

The list of host / index combinations in the lookup is driven from data populated by dbinspect. The logic of the search is "give me the set of dbinspect records from 'today' and count by host, index". If the "today" part of that equation is confused, because of time zone issues, it could result in an empty list. You can always click on the "run" link of the saved searches page (Settings > Searches, Reports and Alerts) for the search in question (here it's "FB - Host Index Cache") to see if it finds results as expected.

[sowings]

Patience, grasshopper! Per my earlier comments, the TA runs itself automatically at midnight. If you installed this morning, you can either wait for midnight, or jumpstart the process with information contained in this post:

http://answers.splunk.com/answers/79629/how-do-i-run-a-scheduled-search-to-refresh-its-cache.html

In this case, you're triggering the "TA-fire_brigade" app's saved search called "Update monitored list from REST" and then "DB inspection". The trick is with the curl command you have to HTML-escape those spaces. To run the first (to update the listing), you can use this URL (with the trigger_actions and everything else included from that above post):

https://<INDEXER>:<PORT>/servicesNS/admin/TA-fire_brigade/saved/searches/Update%20monitored%20list%20from%REST/dispatch

And then the same for "DB%20inspection", replacing the name of the first search with that. From each, you'll get a response that looks like the XML block below. Might want to wait a couple of minutes after the "DB inspection" search to check your dashboards.

Sample output:

<?xml version="1.0" encoding="UTF-8"?>
<response>
  <sid>admin__nobody_VEEtZmlyZV9icmlnYWRl__RMD55aaa2df2491d4b84_at_1438362027_8057</sid>
</response>

[ccsfdave]

Hey I work for government, patience is my strong suit.

I'll check in on Monday...have a good weekend.

[ccsfdave]

I am learning something about myself today, so I wasn't patient. It was like sitting here looking at xmas presents. I couldn't resist. So I ran the curls.

Everything works!!!!

[sowings]

In addition to the main fire_brigade app (which has the dashboards, etc), your indexer needs to have the TA-fire_brigade (technology add-on for Fire Brigade) linked here: https://splunkbase.splunk.com/app/1633/ This piece does the data collection that drives the rest of the app. Without the TA, you'll just have blank dashboards and warnings about empty lookup tables!

[ccsfdave]

Well, installed the TA on the indexer and bounced the indexer and SH. I was thinking, I don't know what a TA will accomplish if the underlying searches are not producing results. I go back to:

| inputlookup fb_hostname_index_cache

and

index=summary search_name="DB inspection"

with no results.

[sowings]

OK! That's a start. Your TA is not collecting the dbinspect data. You've mentioned that you've got a monitored_indexes.csv populated in the main app folder. This one is actually vestigial from an earlier release, and is no longer actually used. You'd actually be looking for the monitored_indexes.csv in the lookups directory of TA-fire_brigade. The latter needs to be installed on all of your indexers. If it's just a standalone all-in-one host, the TA goes on that host, as discussed above.

IF your TA is installed, but is not collecting data, we'd have to check next WHY its not. However, it sounds like you don't yet have the TA installed for data collection.

Earlier versions of the Fire Brigade app did the data collection even in the full app, but starting with 2.0 these functions were decoupled. If you've upgraded from Fire Brigade version 1, this could be what's tripping you up.

[ccsfdave]

I have the fire_brigade app installed on my SH (also a DS) and on the indexer. However monitored_indexes.csv was not on the indexer. So, I SCPed it over, checked the ownership, and restarted the indexer. I reran the index=summary search_name="DB inspection" search but it is still blank.

BTW, my app is fire_brigade not TA-fire_brigade. I am running version 2.0.3 - if any of that is helpful

[sowings]

Try running the "FB - Host Index Cache" search over all time; do you get results? Another way to check for the time ranges of data for the TA's collection is this search

| metadata index=summary type=sources | convert ctime(*Time)

This will ask the summary index (where the TA stashes its results) for the list of sources (in a summary index, the source is the name of the saved search), and pretty print all of the time fields. You're interested in the "lastTime" for the source "DB inspection". Got any? Recently?

Next it might be time zones. Does the server(s) live in a different time zone than your account's setting?

[ccsfdave]

Well, bad news is no results.

I did run the FB - Host Index Cache over all time.

The metadata of sources resulted in nothing for DB inspection.

Everything here is in the same timezone.

Thanks.

[ccsfdave]

So I am trying to trace this back and it appears my fb_host_groups.csv is also blank (except for column names):

cat /opt/splunk/etc/apps/fire_brigade/lookups/fb_host_groups.csv
orig_host,host_group

Now I am trying to figure out how this is populated and through that maybe the dominoes begin to fall?

[ccsfdave]

OK, back to the original response, I looked for the FB - Host Index Cache search and pulled just the first part of it out to run:
index=summary search_name="DB inspection"

It results in nothing returned. I was trying to take the next logical step and figure out what the DB inspection search_name expanded to but the only thing I found was a lookup:

source::DB inspection : LOOKUP-hostgroup

So I am still searching for how to populate the csv... 😞

[sowings]

That's for an optional feature set. Nothing bad will happen if that one is left blank.

If you have a large Splunk environment, you can group the indexers into "host groups" (think data center A, data center B) so that you can get a holistic view of just that group. The overview pages have "host group" counterparts.