Deployment Architecture
Highlighted

Why does an eventtype calling a macro only fails in an indexer / search head cluster? (macro is not expanded)

SplunkTrust
SplunkTrust

Hi !

For a new release of my application, i would have likes to facilitate users customizing by implementing eventtypes with a common simple macro that references index(es) name(s).

Unfortunately, while this works perfectly fine in a standalone instance, this will systematically fail to work in a cluster configuration (indexer cluster and search head cluster)

Am I missing something? Are eventtypes incompatible with the use of a simple macro?

The macros looks like:

# This macro will match nmon index(s) and should be customized if you want to store nmon data elsewhere
[nmon_index]
definition = index="nmon"
iseval = 0

Eventtypes:

####################
# Performance data #
####################

[nmon:performance]
search = `nmon_index` sourcetype=nmon_data

[nmon:performance:cpu]
search = `nmon_index` sourcetype=nmon_data (type="CPU_ALL" OR type="LPAR" OR type="WLM*CPU" OR type="POOLS")

And so on.

--> In standalone mode, no problem, search for data using for example eventtypes=nmon:performance works perfectly fine.

--> In cluster mode, its seems the macro cannot be expanded as long as you access to it through eventtypes.

Example of output logs:

NOTE: the index name is different in the following example:

search eventtype=nmon:performance
over the time range:

2/2/16 12:17:32.000 AM – 2/3/16 12:17:32.000 AM did not return any data. Possible solutions are to:

relax the primary search criteria widen the time range of the search check that the default search indexes for your account include the desired indexes The following messages were returned by the search subsystem:

ERROR: [splunk-peer1] Streamed search execute failed because: nmon_index ERROR: [splunk-peer2] Streamed search execute failed because: nmon_index

Looking at search.log in the job inspector:

02-03-2016 00:17:33.233 INFO  SearchParser - PARSING: litsearch ( ( `nmon_index` sourcetype=nmon_data ) ) | fields  keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"  | remotetl  nb=300 et=1454368652.000000 lt=1454455052.000000 remove=true max_count=1000 max_prefetch=100
02-03-2016 00:17:33.238 INFO  DispatchThread - Disk quota = 10485760000
02-03-2016 00:17:33.240 INFO  UserManager - Setting user context: useradmin1
02-03-2016 00:17:33.240 INFO  UserManager - Done setting user context: NULL -> useradmin1
02-03-2016 00:17:33.246 INFO  SearchParser - AFTER EXPANDING MACROS: litsearch ( ( index="ccr_nmon" sourcetype=nmon_data ) ) | fields  keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"  | remotetl  nb=300 et=1454368652.000000 lt=1454455052.000000 remove=true max_count=1000 max_prefetch=100
02-03-2016 00:17:33.261 INFO  CalcFieldProcessor - Found valid eval expression for field 'OStype' in stanza [nmon_config]': case(OS == "Linux", "Linux", OS == "Solaris", "Solaris", isnotnull(AIX_LEVEL), "AIX", isnull(OS), "Unknown")
02-03-2016 00:17:33.261 INFO  CalcFieldProcessor - Found valid eval expression for field 'cpu_load_percent' in stanza [nmon_data]': (Sys_PCT+User_PCT+Wait_PCT)
02-03-2016 00:17:33.261 INFO  CalcFieldProcessor - Found valid eval expression for field 'hypervisor_id' in stanza [nmon_data]': if(isnotnull(frameID), frameID, serialnum)
02-03-2016 00:17:33.261 INFO  CalcFieldProcessor - Found valid eval expression for field 'mem' in stanza [nmon_data]': case(isnotnull(Real_total_MB), Real_total_MB, isnotnull(memtotal), memtotal)
02-03-2016 00:17:33.261 INFO  CalcFieldProcessor - Found valid eval expression for field 'mem_free' in stanza [nmon_data]': case(isnotnull(Real_free_MB), Real_free_MB, isnotnull(memfree), memfree)
02-03-2016 00:17:33.261 INFO  CalcFieldProcessor - Found valid eval expression for field 'mem_used' in stanza [nmon_data]': case(isnotnull(Real_total_MB), (Real_total_MB-Real_free_MB), isnotnull(memtotal), (memtotal-memfree))
02-03-2016 00:17:33.261 INFO  CalcFieldProcessor - Found valid eval expression for field 'swap' in stanza [nmon_data]': case(isnotnull(Virtual_total_MB), Virtual_total_MB, isnotnull(swaptotal), swaptotal)
02-03-2016 00:17:33.261 INFO  CalcFieldProcessor - Found valid eval expression for field 'swap_free' in stanza [nmon_data]': case(isnotnull(Virtual_free_MB), Virtual_free_MB, isnotnull(swapfree), swapfree)
02-03-2016 00:17:33.261 INFO  CalcFieldProcessor - Found valid eval expression for field 'swap_used' in stanza [nmon_data]': case(isnotnull(Virtual_total_MB), (Virtual_total_MB-Virtual_free_MB), isnotnull(swaptotal), (swaptotal-swapfree))
02-03-2016 00:17:33.261 INFO  CalcFieldProcessor - Found valid eval expression for field 'thruput' in stanza [nmon_data]': case(type=="NET", (value*1000))
02-03-2016 00:17:33.261 INFO  CalcFieldProcessor - Found valid eval expression for field 'converter_inuse' in stanza [nmon_processing]': case(isnotnull(python_version), "Python", isnotnull(perl_version), "Perl")
02-03-2016 00:17:33.261 INFO  CalcFieldProcessor - Found valid eval expression for field 'interpreter_version' in stanza [nmon_processing]': case(isnotnull(python_version), python_version, isnotnull(perl_version), perl_version)
02-03-2016 00:17:33.306 INFO  SearchParser - PARSING: typer | tags
02-03-2016 00:17:33.400 INFO  FastTyper - found nodes count: comparisons=62, unique_comparisons=34, terms=4, unique_terms=4, phrases=12, unique_phrases=12, total leaves=78
02-03-2016 00:17:33.413 INFO  IndexScopedSearch - 0x7f629ae15300 LISPY for index=ccr_nmon is lispy='[ AND sourcetype::nmon_data ]' ct=1454455052 et=1454368652 lt=1454455052 dbsize=0
02-03-2016 00:17:33.414 INFO  UnifiedSearch - Processed search targeting arguments
02-03-2016 00:17:33.437 INFO  LocalCollector - Final required fields list = *,AIX_LEVEL,Message,OS,Real_free_MB,Real_total_MB,Sys_PCT,User_PCT,VP_IdlePCT,Virtual_free_MB,Virtual_total_MB,Wait_PCT,_bkt,_cd,_raw,_si,_subsecond,eventtype,frameID,host,hostname,index,linecount,memfree,memtotal,perl_version,python_version,serialnum,source,sourcetype,splunk_server,swapfree,swaptotal,type,value
02-03-2016 00:17:33.437 INFO  UserManager - Unwound user context: useradmin1 -> NULL
02-03-2016 00:17:33.437 INFO  UserManager - Setting user context: useradmin1
02-03-2016 00:17:33.437 INFO  UserManager - Done setting user context: NULL -> useradmin1
02-03-2016 00:17:33.447 INFO  UserManager - Unwound user context: useradmin1 -> NULL
02-03-2016 00:17:33.493 INFO  UserManager - Unwound user context: useradmin1 -> NULL
02-03-2016 00:17:35.251 INFO  UserManager - Unwound user context: useradmin1 -> NULL
02-03-2016 00:17:35.258 INFO  UserManager - Setting user context: useradmin1
02-03-2016 00:17:35.258 INFO  UserManager - Done setting user context: NULL -> useradmin1
02-03-2016 00:17:35.258 INFO  UserManager - Unwound user context: useradmin1 -> NULL
02-03-2016 00:17:35.273 INFO  DispatchThread - Downloading all remote search.log files took 0.016 seconds
02-03-2016 00:17:35.279 INFO  DispatchManager - DispatchManager::dispatchHasFinished(id='1454455052.19_A4D8E624-DAE2-4B5C-8724-5FE9FADB065A', username='useradmin1')
02-03-2016 00:17:35.284 INFO  UserManager - Unwound user context: useradmin1 -> NULL
02-03-2016 00:17:35.284 INFO  ShutdownHandler - Shutting down splunkd

What you can see here is that the macro initially seems to be expanded:

02-03-2016 00:17:33.233 INFO  SearchParser - PARSING: litsearch ( ( `nmon_index` sourcetype=nmon_data ) ) | fields  keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"  | remotetl  nb=300 et=1454368652.000000 

--> This search will work manually

02-03-2016 00:17:33.246 INFO  SearchParser - AFTER EXPANDING MACROS: litsearch ( ( index="ccr_nmon" sourcetype=nmon_data ) ) | fields  keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"  | remotetl  nb=300 et=1454368652.000000 lt=1454455052.000000 remove=true max_count=1000 max_prefetch=100

The macro has been expanded.

Looking at indexer.log in the job inspector:

02-03-2016 00:17:33.484 INFO  SearchParser - PARSING: litsearch ( ( `nmon_index` sourcetype=nmon_data ) ) | fields  keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"  | remotetl  nb=300 et=1454368652.000000 lt=1454455052.000000 remove=true max_count=1000 max_prefetch=100
02-03-2016 00:17:33.484 ERROR SearchPipelineExecutor - sid:remote_splunk-head1_1454455052.19_A4D8E624-DAE2-4B5C-8724-5FE9FADB065A Streamed search execute failed because: nmon_index
02-03-2016 00:17:33.485 INFO  SearchPipelineExecutor - Finished streaming: results.count=0, sent.bytes.count=2271
02-03-2016 00:17:33.485 INFO  UserManager - Unwound user context: useradmin1 -> useradmin1
02-03-2016 00:17:33.485 INFO  dispatchRunner - Done with streaming search.
02-03-2016 00:17:33.488 INFO  UserManager - Unwound user context: useradmin1 -> NULL
02-03-2016 00:17:33.489 INFO  ShutdownHandler - Shutting down splunkd

It seems that on indexer level, the macro is not being expanded if it called by an eventtype.

eventtypes.conf and macros.conf are deployed on the search head nodes.

For debugging purposes, I have also deployed the macros.conf / eventtypes.conf to indexers, and also set metadata for system share, and re-deploy the cluster bundle.

But this still won't work.

This looks like a bug 🙂

Last but not least, once the macros.conf / eventtypes.conf have been deployed on indexers, I can even search directly on indexers, and the eventtype will work perfectly:

splunk@splunk-peer1:~$ bin/splunk search "eventtype=nmon:performance | head 10"
Your session is invalid.  Please login.
Splunk username: admin
Password: 
DISKBSIZE,splunk-head3,splunk-head3,Linux,60,120,03-02-2016 00:43:20,sda1,13.1
DISKBSIZE,splunk-head3,splunk-head3,Linux,60,120,03-02-2016 00:43:20,sda,12.4
JFSFILE,splunk-head3,splunk-head3,Linux,60,120,03-02-2016 00:43:20,/splunk-logs,90.8
JFSFILE,splunk-head3,splunk-head3,Linux,60,120,03-02-2016 00:43:20,/splunk-SHP,90.8
JFSFILE,splunk-head3,splunk-head3,Linux,60,120,03-02-2016 00:43:20,/splunk,90.8
JFSFILE,splunk-head3,splunk-head3,Linux,60,120,03-02-2016 00:43:20,/run/shm,0.0
JFSFILE,splunk-head3,splunk-head3,Linux,60,120,03-02-2016 00:43:20,/run/lock,0.0
JFSFILE,splunk-head3,splunk-head3,Linux,60,120,03-02-2016 00:43:20,/run,0.5
JFSFILE,splunk-head3,splunk-head3,Linux,60,120,03-02-2016 00:43:20,/dev,0.0
JFSFILE,splunk-head3,splunk-head3,Linux,60,120,03-02-2016 00:43:20,/,65.5
splunk@splunk-peer1:~$ 

But calling the same search from search head will not!

Off course, if I modify eventtypes and replace the macro code with the index name:

 [nmon:performance]
 search = index=ccr_nmon sourcetype=nmon_data

Then no more problem, and eventtypes will work as expected.

Any idea will be welcomed 🙂

Guilhem

0 Karma
Highlighted

Re: Why does an eventtype calling a macro only fails in an indexer / search head cluster? (macro is not expanded)

Splunk Employee
Splunk Employee

Eventtypes are evaluated in distsearch by the 'command.search.typer'. This work will be distributed out to your search peers (indexers), but by default, the macros are not. The search bundle provided by the search head to its peers does not include macros.conf (see distsearch.conf and its [replicationSettings:refineConf] stanza). Typically, a macro is evaluated at the search head before the "remote" (the "map" part of MapReduce) search is sent to the peers.

Put another way, the SH doesn't provide macros by default, since it will expand the macros itself before dispatching the search. In your scenario, the indexers do need the macros to evaluate the eventtypes, so you'll have to whitelist replication of macros.conf.

View solution in original post

Highlighted

Re: Why does an eventtype calling a macro only fails in an indexer / search head cluster? (macro is not expanded)

SplunkTrust
SplunkTrust

Received !

Thank you for your answering 😉

Guilhem

0 Karma
Highlighted

Re: Why does an eventtype calling a macro only fails in an indexer / search head cluster? (macro is not expanded)

Motivator

Hi,

I have deployed macros.conf to Search peers using Cluster master. still , I am not able to see the results of eventtype containing macro.

@guilmxm

could you please let me know how did you whitelisted, because there are two layers to be whitelisted: one in replicationSettings:refineConf and second we need to white list the file in replicationWhitelist. I am little bit confused in doing that.

0 Karma
Highlighted

Re: Why does an eventtype calling a macro only fails in an indexer / search head cluster? (macro is not expanded)

Splunk Employee
Splunk Employee

Macros deployed to the indexers don't get employed by search unless you search directly on the indexer itself. Instead, in a distributed environment, Splunk indexers will employ the "knowledge bundle" as provided by the SH. The problem here is that typically, macros are not provided in the knowledge bundle, as set in distsearch.conf [replicationSettings:refineConf], described above.

0 Karma
Highlighted

Re: Why does an eventtype calling a macro only fails in an indexer / search head cluster? (macro is not expanded)

Path Finder

I've got the exact same problem.

What was your final syntax that worked for you in the distsearch.conf file?

Also did distsearch.conf get installed on the Search Heads, or Index cluster, or both?

thank you!
pj

0 Karma
Highlighted

Re: Why does an eventtype calling a macro only fails in an indexer / search head cluster? (macro is not expanded)

New Member

Hi @guilmxm ,
I hope you're still active in this community since this question is three years old. I am facing the same problem and just don't know further. Do you remember how you solved this and could provide some more information?

Thanks!

0 Karma
Highlighted

Re: Why does an eventtype calling a macro only fails in an indexer / search head cluster? (macro is not expanded)

SplunkTrust
SplunkTrust

HI @at1ll3y !

Active, yes still I am since 2014 😉 (or 2013/2012 maybe... can't remember!)

Never mind, to reply to your question because my goal is providing applications to the community which I have off course no access on the infrastructure, and where one wants the lowest level of complexity for the better features, having any extra complex config required is a no go.

So I am simply providing eventtypes that contain the default index naming convention I use.
A user that needs customisation would simply customise the eventtypes according to their needs.

It is not technically possible to use a macro in the definition of your eventtypes unless you operate some heavy and complexes changes I would not recommend.
Not worth the price in my opinion.

Guilhem

0 Karma