I'd eval a new field for whether it should be displayed or not, with criteria from both. For instance: | eval shouldDisplay=case(sourcetype="sourcetype1" AND someField="y", "true", sourcetype="sourcetype2" AND someField="y", "true", 1==1, "false") | where shouldDisplay="true" ... View more

Take a look at http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/HowconfrepoworksinSHC. Specifically, "The cluster does not replicate any configuration changes that you make manually, such as direct edits to configuration files.". These changes can however be pushed from the deployer. So for instance, on shcluster1, user1 makes a change via GUI. That change is replicated to all shcluster1 search heads. Depending on what kind of changes you're looking for, maybe change to dashboards, you could push that to the deployer in shcluster2, then deploy the change. Maybe speak to what sort of changes you're looking to keep in sync. ... View more

Can you change to the splunk user (su - splunk) when the error occurs and less <file.log> without issue? ... View more

Are the ones that are failing in /var/log ? Can you confirm if you've set permissions there manually or have configured a custom umask on them that will allow "other" or a group the splunk user is part of to read/execute on directories and read files? Also can you confirm that you're not manually setting permissions there when you reinstall splunk, it "just works" after the reinstall. ... View more

Not by default, but you might be able to assign an id to each input, then control via custom css. ... View more

I would definitely investigate mcafee now - i've seen it prevent a lot of applications on windows from working. If it has a log you can view see if anything is getting denied. If you have the ability to disable it temporarily I'd try that. ... View more

This a UF on windows or linux? What's OS / splunk UF version? ... View more

But nothing with LISTENING at the end? You should see an entry like: tcp 0 0.0.0.0:8000 0.0.0.0:* LISTEN So: netstat -an | select-string 8000 | select-string LIST All the entries about fin/close are IPs trying to hit that socket and those connections being torn down. ... View more

I believe with stats you need appendcols not append . For instance: | appendcols [search app="atlas" source="/usr/local/homeaway/atlas-production/logs/*" index="aws_prod_applogs" titan statusCode=200 | stats avg(*responseTime) by date_mday] ... View more

Run command via powershell, not cmd.exe if that's what you were doing. select-string is a powershell command. ... View more

Yes, if you need to re-ingest data on the same host that already ingested them, the fishbucket is keeping track that it already ingested them and won't again. Push your new app that sends those logs to the new location, remove fishbucket entries for that, restart splunk. ... View more

Doesn't seem to be. Your 2nd entry includes AS action , a portion missing from your first. The second one is the correct format, however I don't think it needs windows_action, windows_action twice. ... View more

Has it created a socket on port 8000 yet? Try: netstat -an | select-string 8000 and see if anything returns that it's listening. If not, do you have anything denying it from creating that socket? Anti-virus, etc? ... View more

How many indexers are you spraying data across? You want to spread data evenly across them without overloading one in a cluster. I haven't configured an ALB before, but if you have forwarders in your environment with autoLBFrequency configured (default 30s) in outputs.conf, I would mimic that as a place to start. ... View more

Are you upgrading the app from the left side app listing, or are you going into manage apps first, then upgrading. I noticed that the app name isn't passed along to the upgrade section if you were on the Splunk home page and trying to upgrade from that screen. You'd have to go into manage apps first, then upgrade. Let me know. ... View more

Yea, you can utilized the extracted field from rex, in your next eval. The following should output 1, 2 | makeresults | eval field1="test1" | rex field=field1 "test(?<fieldcount>\d)" | eval field2=fieldcount+1 | table fieldcount, field2 ... View more

I wouldn't think your DS/LM would have KV store listed as a role as that's typically reserved for your search head/search cluster nodes, however that won't affect much. Even removing that, because you're configuring it as a license master it'll throw this warning. It's only a warning and doesn't affect anything. Splunk is just providing guidance that you typically want to designate single roles to your servers. These classifications are only used on the MC dashboards to filter what shows up where. You can safely ignore the message if your server is in fact configured for all those roles. ... View more

How many users do you typically have flagged at one time? How do you go about flagging them in Splunk (just adding to a lookup table maybe)? I haven't worked with ES, but could imagine building something around this with a couple lookups (csv or kvstore) and a scheduled search. One lookup one contain flagged users (update manually or with lookup editor or some other automation), and your scheduled search would constantly scan the last minute, every minute for auth events that match users in that lookup. The search would then input the auth events lookup, find new data in the last minute for matching users, remove users that don't exist anymore, and re-write the lookup. If you're going the ES route, a data model acceleration allows you to use tstats as a fast command to return results. I would assume if you dig into the macros for what's populating your dashboards, you'll find the searches they use behind the scenes. ... View more

Investigate splunkd.log on the forwarder collecting the files for tailingprocessor messages. Also, if each JSON record has the same header (first 256b) then it's possible splunk thinks they're the same file and it's skipping some. In inputs.conf, you can control initCrcLength and crcSalt. Try setting crcSalt=<SOURCE> which will let splunk index files with the same CRC value. ... View more

Did you already verify the connectivity from universal forwarder to indexer (e.g. telnet input-prd-p-g6lslmzlhtlt.cloud.splunk.com 9997, or cat < /dev/tcp/input-prd-p-g6lslmzlhtlt.cloud.splunk.com/9997) ? When you say no data is getting there, did you 1) set up some inputs or 2) check if _internal has any data in it from remote hosts? ... View more

Unix + kerberos in a microsoft active directory environment is tricky. Typically when you see a "server not found in kerberos database" error, you're trying to invoke-command (via winrm) from one windows machine to another, and your trustedhosts config is too restrictive. I'm not positive what the equivalent is on unix, but you might try using kinit first to see if you can request a ticket with the account you're using to connect remotely. That might rule out some problems. For instance: kinit myuser@MY.DOMAIN.COM. If it prompts you for your password, and klist now shows the ticket, things should be in a good state. If your unix host isn't configured to find the domain controllers, you might be running into issues there. I have a similar setup but I think my driver is different. I can confirm after you do some more testing. ... View more

Yea a use case would be nice to have. The problem with constantly updating a csv is you're constantly changing the search knowledge bundle, and I'm not entirely sure what that would do to your environment. A better approach might involve summary indexing, kvstore, or data model + acceleration at the end of the day. I would think constantly outputting a csv would be the last thing you'd want to do. ... View more

While I would highly recommend not upgrading a production environment to a .0 release, if you're going to do it the main concerns are you want to follow the tiered approach (master, search, peers) You don't want peers online with different versions as major releases will introduce different communication methods. So for you, you want to follow the standard cluster upgrade procedure: master, search, master in maintenance mode, then when you get to indexing layer, because multi-site, you want them all off before starting as you're doing a major release. It seems like 7.1 will introduce a better upgrade procedure so maybe this would be your last time doing it exactly this way. The docs indicate when upgrading a multi-site cluster you can go site-by-site but ONLY if you're doing a minor version. Confusing cause they don't list your exact plan to go from 6.6 to 7.1. If I were you, I'd take down all the indexers at all sites, upgrade one site, start it, then move to the next site (again, this is all part of step 3 in the tier'd approach listed). ... View more

Before closing out your </input> area, add: <change> <set token="dropToken_label">$label$</set> </change> Then use $dropToken_label$ as your token for the label of $dropToken$ ... View more

Should be able to just open the python file mentioned in a text editor and save it. Then restart the Splunk service. ... View more