I don't think there's a meaningful difference.  We drop pre-populated etc/passwd files all the time, and it just works.  You do have to restart Splunk for it to see the new passwd file. ... View more

The "query" field is special.  Any field named "query" gets its field name stripped going through format.  Try something like this: | makeresults | eval src_ip="1.2.3.4", src_port=1234, dest_ip="5.6.7.8", dest_port=5678 | eval query=mvappend(src_ip,src_port,dest_ip,dest_port) | fields query | format mvsep="AND"   The last 3 lines are the relevant part, makeresults is just there to help me mock up some stuff. ... View more

Splunk has the existing "| iplocation" command which does this at search time using its local copy of Maxmind's "free" GeoLite2 database. With the correct scripting tools, it's "not difficult" to add your own ranges to the GeoLite2 database and have them show up in the output of the "| iplocation" command just like any external IPs might. Also, a simple XML dashboard to make arbitrary lookups using Splunk's iplocation command is not at all difficult either. In fact, give https://gist.github.com/duckfez/bea451c41be8c2db9fb20647eef4592f a try. ... View more

NOTE I pulled this from a discussion in Slack on this same topic, saving it in Answers for posterity. Sorry about the poor formatting and run on sentences. In AD (and other LDAP tools), both 'users' and 'group' are just objects with attributes. 'Groups' just happen to be objects that have "member" attributes and 'Users' don't. Splunk is relatively flexible about how it handles the mapping of LDAP objects to splunk roles. Take this setup right from the docs: [ldaphost1] host = ldaphost1.domain.com port = 389 SSLEnabled = 0 bindDN = cn=bind_user bindDNpassword = bind_user_password groupBaseDN = ou=Groups,dc=splunk,dc=com groupBaseFilter = (objectclass=*) groupMappingAttribute = dn groupMemberAttribute = uniqueMember groupNameAttribute = cn realNameAttribute = displayName userBaseDN = ou=People,dc=splunk,dc=com userBaseFilter = (objectclass=*) userNameAttribute = uid Notice the groupBaseDN is a different "subdirectory" of the LDAP tree ou=Groups from the userBaseDN in ou=People . This is how most folks have their LDAP trees set up. When you put in a username and password, splunk uses its bind account to run an LDAP search for (&(objectclass=*)(uid=<username>)) inside of ou=People based on the values of userBaseDN , userBaseFilter and userNameAttribute . If this returns a dn that matches (meaning it found you) it will attempt to BIND as that DN with the password you gave. If successful, you are authenticated. Now, to find out what roles you have, it will use its bind account to do another search to map your roles but (again per the above config) it will search in ou=Groups for something matching (&(objectclass=*)(uniqueMember=<your_dn_found_before)) . Whatever "groups" return from this final search, their value of cn (because of groupnameAttribute in the config file) will be matched against the roleMap for that LDAP strategy, and that is what you get in terms of Splunk roles. Now let's compare that to a different example in the docs: [supportLDAP] SSLEnabled = 0 bindDN = cn=Directory Manager bindDNpassword = ######### groupBaseDN = ou=People,dc=splunksupport,dc=com groupBaseFilter = (objectclass=*) groupMappingAttribute = uid groupMemberAttribute = uid groupNameAttribute = uid host = supportldap.splunksupport.com port = 389 realNameAttribute = cn userBaseDN = ou=People,dc=splunksupport,dc=com userBaseFilter = (objectclass=*) userNameAttribute = uid [roleMap_supportLDAP] admin = rlee;bsmith This is weird, but shows off the dynamism of splunk's ldap client. Here, both "users" and "groups" are pulled from ou=People . But looking at the group* settings what we've done is a hack to make "groups" reference "user" objects. The user login will go just like before .. search with the bind account, find you, bind with your dn and password . If successful, then we do the "group" by searching ou=People for someone where (&(objectclass=*)(uid=<your_uid>)) and use the uid value coming back from that in the roleMap lookup -- in effect making your "user" object look and feel like a "group" object. To splunk it's "so I queried the part of the directory tree you told me to, using the parameters you gave me, and I got back results. Whether the users are really users, the groups are really groups, or what, IDGAF" But, these two configurations are mutually exclusive. You can easily configure splunk to map "groups" to splunk roles (see first config). You can easily configure splunk to map "users" to splunk roles (see second config). But, can you configure splunk to map "users" OR "groups" to splunk roles? Taking advantage of the fact that LDAP allows multiple strategies and the first place a user is found is where they will get pulled from we can do something like this: [authentication] authType = LDAP authSettings = ldaphost1-users, ldaphost1 [ldaphost1] host = ldaphost1.domain.com port = 389 SSLEnabled = 0 bindDN = cn=bind_user bindDNpassword = bind_user_password groupBaseDN = ou=Groups,dc=splunk,dc=com groupBaseFilter = (objectclass=*) groupMappingAttribute = dn groupMemberAttribute = uniqueMember groupNameAttribute = cn realNameAttribute = displayName userBaseDN = ou=People,dc=splunk,dc=com userBaseFilter = (objectclass=*) userNameAttribute = uid [roleMap_ldaphost1] user = Splunk_user_group [ldaphost1-users] host = ldaphost1.domain.com port = 389 SSLEnabled = 0 bindDN = cn=bind_user bindDNpassword = bind_user_password groupBaseDN = ou=People,dc=splunk,dc=com groupBaseFilter = (objectclass=*) groupMappingAttribute = uid groupMemberAttribute = uid groupNameAttribute = uid realNameAttribute = cn userBaseDN = ou=People,dc=splunk,dc=com userBaseFilter = (&(objectclass=*)(|(uid=rlee)(uid=bsmith)) userNameAttribute = uid [roleMap_ldaphost1-users] admin = rlee;bsmith Notice these two "strategies" point to the same LDAP server, just queried in different ways -- same bind account and everything. When you have multiple LDAP "strategies" (stratagems?), then the first one to find a user is the one used. In this config I explictly search the ldaphost1-users strategy first, and have made it treat users and groups as basically identical objects (as shown in the docs link above) so that I can map user names to roles. But, I've nerfed the userBaseFilter in such a way that it will only ever find people who are mapped to a role below. If you're not in the filter for the first strategy, then it falls through with "no user found", and goes to the second one that queries the same LDAP server -- but this time, it maps groups to roles and proceeds from there. While this should work, this is terrible for the "users directly get a role case" because you have to both update the role map and the filter each time you want to add a user, and the filter will get unwieldy fast. Also there's another interesting side effect of this config -- if you map a user to a role in the ldaphost1-users stanza, then all of their role mappings must be done there. You can't "also include them in a group" and expect that to work. ... View more

The si commands - sitimechart , sitop , sistats etc all save data in what is basically a serialized format called colloquially the "prestats" format. You get new fields that are named with things like psrsvd_X - these contain the "sufficient statistics" needed to make a summary index work. Depending on the functions in your stats (or timechart - remember that timechart is really just "stats with a funny hat") command, it may need to save different things in order to reconstruct a final result over your summary. For example, if you're summarizing count() or sum() then it's pretty each to just store the number or items counted, or the sum of them. The sum of a set of sums is still something you just add up. But, average is different. If Splunk stored the average at each "sample" of the summary index, then averaged-the-averages you'd get a very close but not mathematically accurate summarization. Instead, Splunk stores what the average is made of -- a count and a sum. For some experimentation and an example, I'm using this metrics log data everyone has: index=_internal source=*metrics.log group=queue | sitimechart span=1h avg(current_size_kb) by name When I run this, I get a result with some funny psrsvd fields in it, like psrsvd_ct_current_size_kb and psrsvd_sm_current_size_kb . There are some others too, but I want to focus on these two. These are, during the generation of the summary, the count and the sum. Let's do some experimentation: index=_internal source=*metrics.log group=queue | sitimechart span=1h avg(current_size_kb) by name | search name=indexqueue | search psrsvd_sm_current_size_kb > 0 | eval average=psrsvd_sm_current_size_kb / psrsvd_ct_current_size_kb I am taking the output of my sitimechart and manually recreating the definition of average by dividing sum by count. Then we'll compare it to basically the same data in timechart . Now I'll run a very similar, but not identical, search using just timechart . index=_internal source=*metrics.log group=queue name=indexqueue | timechart span=1h avg(current_size_kb) by name | search indexqueue!=0 Cool, the exact same averages. Part of the point here is that my summary index generation parameters and my summary index usage parameters can be a little different and the maths still work out to be reasonable. Like, if I'm collecting CPU usage on an hourly average, then I can report on it with an hourly average, or a daily average, or a weekly average or whatever -- it's just a question of how many different values of "count" and "sum" that I add up in order to compute an average. (Note this is only true when you roll up to larger time windows - I can't take an hourly sample of average CPU and use that to derive a per-minute average). This helps you understand why Splunk stores summary index data in this strange prestats serialized format - its more flexible for reporting on later, even if it's not easily human readable. While I was writing this, I set up a summary index to run as follows: index=_internal source=*metrics.log group=queue | sitimechart span=1m avg(current_size_kb) by name This is in a scheduled search named "test", scheduled for every hour over the past hour. If I look at the data in the summary index, index=summary search_name="test" I see it's in the prestats format, like so: 06/17/2018 12:14:00 -0500, search_name=test, search_now=1529255700.000, info_min_time=1529252100.000, info_max_time=1529255700.000, info_search_time=1529255700.170, name=udp_queue, psrsvd_ct_current_size_kb=2, psrsvd_gc=2, psrsvd_nc_current_size_kb=2, psrsvd_sm_current_size_kb=0, psrsvd_v=1, psrsvd_vt_current_size_kb=0 Now I can use a "regular timechart" to get data out of the summary and present it in a sane visualization: index=summary search_name="test" | timechart span=1h avg(current_size_kb) by name Hopefully this helps as a walkthrough of what is happening and lets you figure out where you've got things askew. ... View more