We're summary indexing events from one index into another.  The original index contains JSON events e.g. {"field1": "value1", "field2": "value2"} The summary events all have a hostname value appended onto them e.g. {"field1": "value1", "field2": "value2"}, hostname=abc I found abc configured in alert_actions.conf under the email stanza; [email] hostname = abc I can't figure out why this is happening and how to remove it from future summary events.  In advanced edit I can see a field called action.summary_index.hostname which contains "abc".  Is it simply inheriting the hostname parameter from alert_actions.conf? I tried removing the value from this parameter and saving but when I go back into advanced edit the value is back.  This search cluster is quite mature (old...) so I wouldn't be happy simply removing that hostname parameter from alert_actions.conf and "hoping for the best".  I'm not sure where else it might be being used and what impact it could have on other users. Any thoughts at all would be welcome here. ... View more

I've just finished adding new physical indexers to our existing multi-site indexer cluster and I'm trying to figure out the safest method for removing the old virtual indexers. We started off with a 2 site cluster with each site having 3 members and the following config. available_sites = site1,site2 multisite = true replication_factor = 2 site_replication_factor = origin:1,total:2 site_search_factor = origin:1,total:2 When I added the new indexers I created 2 new sites (3 and 4) and amended the config as follows. available_sites = site1,site2,site3,site4 multisite = true replication_factor = 4 site_replication_factor = origin:1,total:4 site_search_factor = origin:1,total:4 Now that the upgraded cluster has equalized I'm trying to figure out what the safest method for removing sites 1 and 2 is.  I think it should be; 1. splunk offline --enforce-counts (while watching the indexer clustering dashboard on the CM waiting for all data to be searchable before offlining the next). 2. Put the cluster into maintenance mode and update server.conf on the CM as follows; available_sites = site3,site4 multisite = true replication_factor = 2 site_replication_factor = origin:1,total:2 site_search_factor = origin:1,total:2 3. Disabled maintenance mode. Any and all thoughts/past experiences appreciated. ... View more

I've been investigating this error which is appearing 6 times per search (1 for each indexer in the cluster) for a while now and have come to an end pass... This is happening for all non Admin roles. The permissions are global (all apps) for both the lookup file and the definition with everyone having read access to both. This is all configured on the search head cluster. The actual lookup csv file exists on all search heads in the search head cluster and are owned by splunk:splunk. I read on a few other questions that the lookup definition should also be deployed to the indexer cluster but I'm not sure why this would matter. In my case this error is appearing for searches that aren't even referencing the lookup in question. Do I really need to configure and schedule the search which creates the lookup for the search head cluster on the indexer cluster as well? search.log on the search head 09-11-2017 13:15:42.187 WARN LookupOperator - Unable to find property=filename for lookup=XXX will attempt to use implicit filename. 09-11-2017 13:15:42.187 WARN LookupOperator - No valid lookup found for lookup=XXX 09-11-2017 13:15:42.187 ERROR LookupOperator - The lookup table 'XXX' does not exist. It is referenced by configuration 'YYY'. 09-11-2017 13:15:42.187 ERROR LookupOperator - The lookup table 'XXX' does not exist. It is referenced by configuration 'YYY'. search.log from an indexer 09-11-2017 13:15:42.338 WARN LookupOperator - Unable to find property=filename for lookup=XXX will attempt to use implicit filename. 09-11-2017 13:15:42.338 WARN LookupOperator - No valid lookup found for lookup=XXX 09-11-2017 13:15:42.338 ERROR LookupOperator - The lookup table 'XXX' does not exist. It is referenced by configuration 'YYY'. 09-11-2017 13:15:42.338 ERROR LookupOperator - The lookup table 'XXX' does not exist. It is referenced by configuration 'YYY'. Has anyone got any ideas on what could be causing this or where to look next? ... View more

I've a request from the business to produce a chart with multiple series which usually isn't a problem but in this case they've added some complexity by asking for weekly averages on top of daily totals so I'm trying to work with multiple bins with different sizes. Series1 - Columns showing the number of daily transactions for the last 30 days Series2 - A line showing the average daily response time for those daily transactions Series3 - A weekly average line showing the average response time Mon-Fri The first 2 are easy but I'm having trouble trying to figure out the last bit. The Statistics tab SHOULD look as follows assuming the 1st and 8th are Mondays; 01/08/2016 - 150 - 2.5 - 2.8 02/08/2016 - 124 - 3.1 - 2.8 03/08/2016 - 167 - 2.1 - 2.8 04/08/2016 - 167 - 4.1 - 2.8 05/08/2016 - 167 - 1.3 - 2.8 06/08/2016 - 167 - 2.1 - 2.8 07/08/2016 - 167 - 4.4 - 2.8 08/08/2016 - 150 - 2.5 - 2.7 09/08/2016 - 124 - 3.1 - 2.7 10/08/2016 - 167 - 1.9 - 2.7 11/08/2016 - 167 - 3.2 - 2.7 12/08/2016 - 167 - 2.9 - 2.7 13/08/2016 - 167 - 2.1 - 2.7 14/08/2016 - 167 - 3.2 - 2.7 I tried building up the search as follows using appendcols to add the weekly averages data. index=myIndex sourcetype=sometype source=somesource | bin _time as TIME span=1d | convert ctime(TIME) | stats avg(responseTime) as AvgDailyResponse, count as CallsPerDay by TIME | appendcols [search index=myIndex sourcetype=sometype source=somesource | bin _time AS TIMEW span=1w | convert ctime(TIMEW) | stats avg(responseTime) as AvgWeeklyResponse] I'm probably close but just can't finish it off. When I run this search I get a new column called AvgWeeklyResponse that only has a value in the first row rather than weekly averages across the days of each week. ... View more