It sounds like you're new to this environment? Are you sure you have an indexer cluster? If you do, your indexers would know which server the master is. And like renjith said, it would be in the clustering stanza in server.conf. So maybe run btool on one of your indexers and see if it's a slave to a master. This should return any non-default settings for the clustering stanza, and which files those settings are coming from. /opt/splunk/bin/splunk btool server list clustering --debug | grep -v system/default ... View more

That does look like a fun log file! I do see group by in there. Is this log in Splunk? But you can find it by searching for "group by"? I understand that if those are all separate events, getting just a group by event back may not be too helpful. But I wonder if maybe it's worth putting all of those script events into one large event before indexing? Not sure how easy/feasible that would be, but could be worth a shot. Also, you mentioned wanting to know which user ran that. Is that in the log anywhere too? ... View more

interesting...maybe it's in limits somewhere. What about looking up from the lookup? Something like this? | inputlookup mdm_master | tstats max(MDM_Last_Check_in_epoc) as MDM_Last_Check_in_epoc | lookup mdm_master MDM_Last_Check_in_epoc | table MDM* | outputlookup mdm_master ... View more

Not sure if you tried Ryan's search,but you could create a lookup in splunk with your list of servers - just a simple csv file with one column and header named host, for example. Call it servers.csv? Then you could run a search to join that list with servers splunk knows about. Something like this (untested) | inputlookup servers.csv | join type=left host [|metadata type=hosts ] | where isnull(lastTime) So the inpulookup command will essentially return your csv file, which is just a list of hosts. And the metadata command will return some basic info about all of the hosts splunk knows about, including the last time it saw something from a host. If you do a left join, then you'll keep everything from your lookup and only those items from metadata that match. So if any of the servers in the list don't have the fields that come with the metadata command, like lastTime, then they may have been missed. ... View more

Sorry for the slow response, I don't have a lot of time to spend out here while at work. What about an example where the script data is included? Is it similar but with a script section? ... View more

either should be fine, I think. Are your events being parsed correctly in Splunk? Meaning, do the log files look broken up correctly in Splunk? Or do you have say multiple events in Splunk that you would consider to be one entry in the log file? ... View more

Understood. Are you able to post some of the sample log entries, of course masking anything private? We'll need some idea what the data looks, otherwise we really won't know how to help you search it. ... View more

There is some documentation on Splunk's site that might be useful: http://docs.splunk.com/Documentation/Splunk/6.4.1/AdvancedDev/AppIntro An app is essentially just a container for other objects - reports, dashboards, lookups, field extractions, etc. And if you have the permissions, it should be a simple as creating a new app from the Splunk web interface (to get started at least). But it helps to have a good understanding of how Splunk works as well. You may be able to copy an app, but that could tricky/frustrating if you don't know how the app works, and can't figure it out. You might also be able to create a custom dashboard in the app itself as well. That way you can take advantage of its field extractions, macros, etc. ... View more

can you explain the logic behind the search results you want? how did you get those final numbers based on the first set of numbers? are any of those fields already extracted in your data? Does each event in Splunk contain the timestamp and the a1-a4 lines? ... View more

I think some sample log entries would be helpful. For example, does adding "group by" to your search bring back entries you want or are the log files more complicated than that? But most likely, splunk can get at the data you want, yes. We just need to better understand what that data looks like. ... View more

I believe there is an add-on that needs to go on the parsing layer of your env (probably the indexer). There's a chart on this page of the documentation that highlights which app/add-on needs to go where. http://docs.splunk.com/Documentation/MSApp/1.2.1/MSInfra/HowtodeploytheSplunkAppforWindowsInfrastructure ... View more

Ok, looks like we're getting closer, maybe? If you run the subsearch on its own, without the call to stats (so just the timechart), do those counts look right? And if so, does adding the stats max command back in work correctly? I feel like I'm missing something silly here... ... View more

Ok, I think we're on the same page. This is the subsearch above that was throwing me off, because it just gets a total count of events over the past year. [search index=ast sourcetype=poc_agi_logs agi_dnid=* grp=* earliest=-1y latest=-1d |stats count as "HWM" |eventstats allnum=true max("HWM") AS "High Water Mark"] That said, I thought the search in my answer was doing what you wanted. I don't know why it's not returning any results. Does the subsearch return what looks to be a correct hwm at least? I think that should get a counts for every day over the past year and the grab the max value. ... View more

Hmm, ok. I ran this same search against a simple _internal data set and did return results. Not sure what is different with your data? Did you run it over 30 days like you wanted? Can you run either the outer search or subsearch on their own? For HWM, I understand the concept. But a max value of what over which period? It seems like you're just comparing counts of events (calls) essentially. But your timechart over 30 days will be counts per day. The subsearch in your original post is counting events over an entire year. The comparison doesn't make sense. I understand if your hwm mark intended to represent the count during the day in the past year that had the most calls, but I can't figure out if that's what you're actually asking....but that's what I was trying to answer. ... View more

I'm not quite sure what the hmw is supposed to represent, because it looks like it's just a count of events over the past year (up until yesterday). Is that what you want it to be? The count in your 30-day timechart will be a count by day (or whatever span Splunk uses). So is the goal to have the hwm use the same span as the 30-day timechart (so about 12 different count values). And from those values you want the max? In general though, I don't think you're using subsearch way you want. If you look at the job inspector, it's likely just adding another condition to your base search for a "High Water Wark" field, which I don't think exists in your dataset?. In other words, that subsearch isn't adding to data to your search results, it's trying to filter the initial results. Assuming I understand what you're looking for, something like this way work a little better, but still feels a bit "out of the way", so there's probably a better approach yet. index=ast sourcetype=poc_agi_logs agi_dnid=* grp=* | timechart count as "Total Calls" | appendcols [ search index=ast sourcetype=poc_agi_logs agi_dnid=* grp=* earliest=-1y latest=-1d | timechart span=1d count | stats max(count) as "High Water Mark" ] | filldown "High Water Mark" So build your timechart, then append a column that represents the hwm from last year. But since that just adds it to the first event, fill that down through the rest of the events. ... View more

Oh, whoops....you can use sort 0 to remove the default limit, like | sort 0 mdm_mac_server - MDM_Last_Check_in_epoc ... View more

I'm not sure if this is what you're looking for, but it seems like the whitelist filter will not include servers by default. They have to be specified in whitelist entry. But the blacklist filter would blacklist by default. http://docs.splunk.com/Documentation/Splunk/latest/admin/Serverclassconf filterType = whitelist | blacklist * The whitelist setting indicates a filtering strategy that pulls in a subset: * Items are not considered to match the stanza by default. * Items that match any whitelist entry, and do not match any blacklist entry are considered to match the stanza. * Items that match any blacklist entry are not considered to match the stanza, regardless of whitelist. ... View more

Like others have alluded to here, you need to elaborate on the scenario you're trying to account for. If you're concerned that you may have completely missed installing a forwarder on some of the servers needed for the project, then you need to have a list of expected servers somewhere to create a lookup from. Unless your servers are named in a way that you can logically determine which ones should be there, e.g. server00, 01, 02 ....37, splunk isn't quite omniscient yet 🙂 If you're concerned that of the forwarders you installed that maybe some are configured right or not sending the project logs, then I think some of the other suggestions here will work. ... View more

This certainly isn't my wheelhouse, but you might want to elaborate a bit on situation. Is this a single server environment? Which conf file was overwritten? How do you plan to get it back by downgrading? What is the actual state of your environment at the moment? I mean, sure, you can uninstall 6.4 and install 5.0. But without knowing what problems you're actually trying to address from the upgrade, I imagine it will be hard for others to advise you on whether those steps make sense. ... View more

Not sure how these things work under the hood, so this may be as or more inefficient, but you could try using sort/dedup. | inputlookup mdm_master | eval mdm_mac_server = MDM_MAC_Address." ".MDM_Server | sort mdm_mac_server - MDM_Last_Check_in_epoc | dedup mdm_mac_server | table MDM* | outputlookup mdm_master Also, again not sure how efficient, but you could try to use a stats instead of eventstats, and then lookup or join the results to the lookup again, keep the ones that match output the lookup again? ... View more