I like @chuckers suggestion that you have a second column, but for a different reason. My original answer works great if there aren't too many domains (less than a hundred). But if there are more, you need a different approach. Given a lookup CSV file like this domain,flag company.com,0 comp2.com,0 etc Upload the file to Splunk and set up a lookup, which I will call domain_lookup. Be sure to set a "default" for the lookup of "Unknown" - this is what will be returned if there is no match. Then search like this index=dns | lookup domain_lookup domain OUTPUT flag | where flag = "Unknown" This will eliminate any events that have the matching (white-listed) domains.This solution will work well no matter the number of white-listed domains. It will out-perform other solutions that use append or sub-searches. Again, the above solution assumes that the field in the events is also called domain and matches the format of the lookup file. If the events in the dns index have a different format, you will need to insert steps in between lines 1 and 2, to create a field named domain that matches the lookup format. ... View more

First, a CSV file that is used for lookup must have a header row. Assuming that the header is simply "domain" - then the field name for the lookup is "domain" and the following will work index=dns NOT [ inputlookup whitelist_domains.csv] Note that the domain field in the CSV must exactly match the field name and the field format in order of this to work. ... View more

I am struggling a bit with the timeline chart - working on it... ... View more

If all of your "search queries" are the same (or even substantially similar), this could be a very inefficient search. If they are all the same: search query (check.status = 2 OR check.status=0) | stats count as occurrences min(timestamp) as minTime by id, client.name, client.address check.status | where occurrences > 2 | eval first_time=if(check.status==2,minTime,null()) | eval recov_time=if(check.status==0,minTime,null()) | stats first(first_time) as first_time first(recov_time) as recov_time by id, client.name, client.address check.status | eval total_time = recov_time - first_time | rename total_time as "Total Time (in secs.)" client.cluster as "Cluster Name" client.name as "Host Name" client.address as "Host Address" | eval "Recover Time"=strftime(recov_time,"%Y/%m/%d %H:%M:%S") | eval "First Seen Time"=strftime(first_time,"%Y/%m/%d %H:%M:%S") | table "Cluster Name","Host Name", "Host Address", "First Seen Time","Recover Time","Total Time (in secs.)",id, occurrences ... View more

In the props.conf (on the search head or wherever users log in), add the following: [xyz] KV_MODE = auto And the key-value pairs (separated by 😃 will be extracted. You do not need the transforms.conf. ... View more

This is not part of the field name. The "#" means that Splunk considers the field numeric, and the "@" means the field is alphanumeric. ... View more

You set the maxresultrows to 5 million, when Splunk specifies that it cannot be more than 50 thousand. From the documentation: * This limit should not exceed 50000. Setting this limit higher than 50000 causes instability. * Defaults to 50000. It is a wonder that your Splunk did not simply crash. What are you trying to do where over 50K of raw events will be useful? That is many more events that anyone could possibly examine. If you categorize your data, Splunk will be able to produce up to 50K rows of results, which could summarize your "lakh" events. Splunk can search all those events, but it cannot return that many results to the browser!! ... View more

Is there anything that identifies the second event as being an answer to the challenge question? ... View more

Try this yoursearchhere | stats latest(LastScanDate) as MostRecentScanDate count by Name | where count = 1 | table Name MostRecentScanDate This produces a list of servers with their most recent scan date. It only shows servers with one entry, which should be "new" servers. I am unclear what you wanted to plot with the timechart command, so I kept it simple here. ... View more

Of all the answers, I like @mmodestino's best - tstats is fast and accurate. metadata can return incomplete results in a larger/busy environment, and the normal index=* search is slow. All of these solutions use the host that is supplied by the parsing process. Imagine that you have a server that is a log repository - perhaps it is a file server with log files from a bunch of different machines. When you set up the forwarder, you should properly set the host to the name of the originating machine. And the name of the originating machine will be used in the tstats or whatever. That's great if you want to measure how much data / which sourcetypes / etc. is coming from each original host. But if you want to measure forwarder traffic, look at the Monitoring Console. Or make your own search, perhaps something like this: index=_internal sourcetype=splunkd component=LicenseUsage | stats sum(b) as b by idx host h s st | rename host as sourceServer | rename h as host s as source st as sourcetype idx as index | eval MB = round(b/1024/1024,1) | fields - b ... View more

What you want is pretty easy to calculate, although I don't know if it makes sense to sum the cpu and memory in this way... But the hard part is formatting... it is quite difficult to produce this exact format in Splunk. yoursearchhere | bucket _time span=1d | stats sum(mem) as daily_mem sum(cpu) as daily_cpu by devname _time | eventstats sum(mem) as totalmem sum(cpu) as totalcpu by devname | where totalmem > 80 or totalcpu > 80 | fields - totalmem totalcpu | eval daily = daily_mem . " " . daily_cpu | eval date = strftime(_time,"%x") . "\nmem cpu" | xyseries devname date daily The alignment of the data may need to be tweaked, and it may never be perfect. To make the reports separately is easier; here is one for just memory yoursearchhere | bucket _time span=1d | eval date = strftime(_time,"%x") | stats sum(mem) as mem by devname date | addtotals | where total > 80 | fields - total ... View more

Splunk expects the lookup files to be in the UTF-8 character set, with normal line endings (Linux or Windows). Here are the specific requirements from the Configure CSV lookups section of the Knowledge Manager manual. The file must also be in proper CSV format. Many text editors can find and "zap" weird characters and clean up the line endings in a file. I think Notepad++ may do this, as will BBEdit and others. ... View more

I am not sure about the trial version, but I know that in the full Splunk Cloud, customers cannot install apps without coordinating with the Cloud Support Team. Since Splunk is making some contractual guarantees about uptime/performance, the use of apps is pretty carefully managed for the customers. If you want to play around with the apps, the easiest way is to create a small Splunk Enterprise trial instance. There you can load any apps that you like for testing. ... View more

What kind of license are you using? What version of Splunk are you running? ... View more

Yes, the specific indexer is getting this issue for one of the following reasons: 1 - the indexer should be connected as a slave to a license master in your environment, but it is not. 2 - the indexer does not have a current license installed. 3 - the indexer has exceeded its daily indexing quota too many times in the last 30 days. If you log onto the indexer as an admin, you can see (and change) the license information under Settings>>Licensing In the Splunk Administration manual, there is a section on configuring Splunk licenses. The manual also explains how license violations work. You may need to contact Splunk Support to unlock the search functionality. ... View more

IMO fieldformat is the unloved stepchild of Splunk commands. My suggestion is to avoid it as much as possible. It is a great idea to separate the value of a field from its displayed representation - but fieldformat doesn't do a very good job of this. I have investigated its behavior for years and across versions and the nicest comment I can make is "it isn't consistent." In dashboards, if you possibly can, use the new numeric formatting; it's nicer, it's efficient, and it works well. Otherwise, I would use eval. ... View more

If you want, you could modify this search to include hosts without an entry in the lookup table - see below. This method still allows you to explicitly set some hosts for no monitoring as well. | tstats last(_time) as lastSeen where index=* by host | append [ inputlookup host_settings.csv ] | stats last(lastSeen) as lastSeen last(monitor) as monitor last(threshhold_minutes) as threshhold_minutes by host | where isnull(monitor) OR monitor="Y" | eval threshhold_minutes=if(isnull(threshhold_minutes),30) | eval status=case(isnull(lastSeen),"MISSING", lastSeen >= now()-(threshhold_minutes*60),"okay", 1==1,"MISSING") | eval lastSeen = strftime(lastSeen,"%x %X") | table host lastSeen status threshhold_minutes This is a pretty good idea in general (I hadn't thought of it before) - in case you have some new hosts in your environment but forget to add them to the CSV. Now they will always be monitored with the default threshhold of 30 minutes if they don't appear in the host_settings.csv ... View more

Yes, absolutely. The DBConnect app is for collecting data from inside a relational database, not for collecting the database logs. For any log file (or other text file), you can use the Splunk input settings to read the file; you don't need an app for that. If you want to test a particular log file, you can copy it to your Splunk server and index it as a local file. Once you are good with that, you can use a forwarder with the same inputs.conf to collect the live logs and send them to your Splunk server. Again, no apps are required. However, sometimes you will find that there is an app on Splunkbase (www.splunkbase.com) that will do exactly what you want: collect the log files, parse them and provide some interesting reports. So it's always worth checking but it is never required. ... View more

Ah - that's helpful to know. Have you looked at the splunkd.log for any error messages? ... View more

You could certainly setup a lookup file for your hosts, perhaps something like this: host_settings.csv host,monitor,threshhold_minutes ahost,Y,30 bhost,Y,10 hostc,N,0 Your search: | tstats last(_time) as lastSeen where index=* by host | append [ inputlookup host_settings.csv ] | stats last(lastSeen) as lastSeen last(monitor) as monitor last(threshhold_minutes) as threshhold_minutes by host | where monitor="Y" | eval status=case(isnull(lastSeen),"MISSING", lastSeen >= now()-(threshhold_minutes*60),"okay", 1==1,"MISSING") | eval lastSeen = strftime(lastSeen,"%x %X") | table host lastSeen status threshhold_minutes I hope this gives you a good starting point. Why am I appending instead of searching with the lookup table? For my purposes, I want to create a list of all the hosts, whether they have had data within the search time period or not. (Set your search time to approximately the longest time you want to monitor, in your example: 60 minutes.) Why use tstats instead of metadata? tstats is very fast, almost as fast as metadata. The metadata command however, can return partial results in larger environments. So if you want better accuracy in this case, use tstats. If not, then just change the tstats to metadata and proceed... ... View more

Is this a Splunk problem, or is it that you are using Splunk to detect the problem? If it is actually a hadoop/yarn problem, then this is not the forum for that question - although it is possible that someone here might know the answer... ... View more