One option would be to run Splunk as a non-root user, and set the umask on the Splunk account. You would need to make sure all log files are viewable by the Splunk account, of course. If you do need to run Splunk as root, POSIX ACLs may be your answer. Something like the following: # Make sure all existing files are group-owned by Splunk. Set the filesystem # ACL to allow the Splunk group as the default for new files, and set the # same ACL on currently existing files. cd /opt/splunk/etc/apps sudo setfacl -R -b . sudo chown -R splunk:splunk . sudo setfacl -R -d -m g:splunk:rwx . sudo setfacl -R -m g:splunk:rwx . # Then repeat the above for /opt/splunk/etc/system/local or other paths as desired. ... View more

Not sure I follow. Are you expecting to see a difference in the log entries themselves? The lookup values appear as new extracted fields, so you should start to see them in the field picker at the left. You might need to click pick fields to bring up the full list. ... View more

The main thing is to make sure your sourcetype is set to ias . ... View more

More detailed instructions are now in a separate Answers post. http://splunk-base.splunk.com/answers/42717/how-do-i-enable-remote-agent-management-in-splunk-for-ossec ... View more

The timeout was increased to 30 seconds in version 1.1.89. ... View more

This error can also be caused by a bug that has been identified in version 1.1.88, which should be fixed as of 1.1.89. ... View more

Warnings: By following the steps below, you will be granting the Splunk service account access to log into your OSSEC server and run commands. Be sure you fully understand any security implications for your environment before proceeding. In particular, enabling the MANAGE_AGENTS option will allow anyone with sufficient access in Splunk to see and modify your OSSEC agent keys. Examples below assume the default configuration (Splunk running as root and installed in /opt/splunk, OSSEC installed in /var/ossec), and that you will use an account named splunk to log into the OSSEC server. Basic familiarity with unix and OSSEC is assumed. Basic commands to log in and out, etc. are not shown. Remote Access Configuration: First, you will need to make sure that the Splunk server can log into the OSSEC server to run management commands. On the OSSEC server, create a new login account for the Splunk server to use when connecting. root@ossec_server$ useradd splunk On the Splunk server, create an SSH keypair for the root user (or whichever account splunkd is running as), and copy the public key to the OSSEC server. user@splunk_server$ sudo su - root@splunk_server# ssh-keygen root@splunk_server# scp .ssh/id_rsa.pub splunk@ossec_server:authorized_keys On the OSSEC server, log in as the splunk account and configure the authorized_keys file to allow SSH logins without a password: splunk@ossecserver$ mkdir .ssh splunk@ossecserver$ mv authorized_keys .ssh/ splunk@ossecserver$ chmod -R go-rwx .ssh Verify that the Splunk server can log into the OSSEC server without a password prompt. You MUST do this at least once and say yes to the SSH key prompt. The second run should not prommpt. root@splunkserver# ssh splunk@ossec_server splunk@ossecserver$ exit root@splunkserver# ssh splunk@ossec_server splunk@ossecserver$ exit On the OSSEC server, configure sudo to allow the splunk login account to run agent management commands without prompting. root@ossecserver# /usr/sbin/visudo (Add the following two lines): splunk ALL=NOPASSWD: /var/ossec/bin/agent_control -l splunk ALL=NOPASSWD: /var/ossec/bin/manage_agents On the OSSEC server, verify that the new splunk account can run the agent management commands without prompting. If either of the following commands prompts for a password, you may have made a mistake in the previous step: splunk@ossec_server$ sudo /var/ossec/bin/agent_control -l splunk@ossec_server$ sudo /var/ossec/bin/manage_agents On the Splunk server, verify that you can remotely run the commands without a password: root@splunk-server$ ssh ossec-server -t -l splunk sudo /var/ossec/bin/agent_control -l root@splunk-server$ ssh ossec-server -t -l splunk sudo /var/ossec/bin/manage_agents App Configuration: All of the following steps are performed on the Splunk server. Check to see if you already have a local copy of ossec_servers.conf: root@splunk_server# cd /opt/splunk/etc/apps/ossec root@splunk_server# ls -l local Create the local directory and ossec_servers.conf file if they are missing: root@splunk_server# mkdir local root@splunk_server# cp default/ossec_servers.conf local/ Edit local/ossec_servers.conf and disable the local machine if you do not have an OSSEC server on the local machine. [_local] DISABLED = True In local/ossec_servers.conf , add your new server: (If your ssh key is in the default path, the '-i' parameter used in some examples is not reuquired) [ossec_server] AGENT_CONTROL = ssh ossec-server -t -l splunk sudo /var/ossec/bin/agent_control MANAGE_AGENTS = ssh ossec-server -t -l splunk sudo /var/ossec/bin/manage_agents Final Steps: Optional: Restart the Splunk daemon to force the scripted inputs to run immediately. /opt/splunk/bin/splunk restart splunkd Optional: In the Splunk for OSSEC app, run the saved searches to regenerate lookup tables. Run each of the following searches from the menu under Searches & Reports -> Utility: OSSEC - Initialize Server Lookup Table OSSEC - Rebuild Server Lookup Table ... View more

There's a good chance this is caused by your data collection timing out, and having the field match on partial context in the timeout error message. For more detail, see the response to your other posted question:      http://splunk-base.splunk.com/answers/29021/ossec_agent_statuspy-v-on-local-server-timeout-exceeded-error ... View more

Timeout issue The most likely reason for the timeout is just the relatively large number of agents - the default timeout is 5 seconds. I've made a note to increase that and/or make it configurable. In the mean time, try editing bin/pyOSSEC.py. At line 331, change: p = pexpect.spawn(cmd, timeout=5) to: p = pexpect.spawn(cmd, timeout=30) and see if that solves the timeout concern. Truncation Issue There's a good chance that the truncation is caused by the timeouts. The partial output from pexpect will print out 100 characters of context for diagnostic purposes. In the example you posted above, it's the section that looks like this: before (last 100 chars): pvap020, IP: 10.180.5.151, Active ID: 1036, Name: w1pvap003, IP: 10.180.5.152, Active ID: 10 Here, it truncated what would have been the ID value. If the truncation had occurred toward the end of a line instead of toward the beginning, you would get something ending with a partial word that would be extracted into the Status column. It might still be something else, but the best approach will be to fix the truncation issue and look more closely at this if the problem persists. If you search on sourcetype=oseec_agent_control , I would expect to see see lines where the raw data shows the same truncation. (Assuming that's the case, it supports the conjecture that it's a data collection problem and not a field extraction problem). ... View more

+1 to araitz's answer. That's similar to our in-house setup. I've uploaded a copy of what we're already using to SplunkBase. As it may take a day or two to get through their approval process, you can also download a copy here: http://www.southerington.com/redir.php?id=273 ... View more

Discrete hours (i.e., 1-2 pm), or on a sliding window? The former is easier than the latter. ... View more

In that case, your stanza header in props.conf would look like [source::.../access_servername.log] . The rest should match up with what is in the documentation link given above. The TRANSFORMS-xxx line in props.conf tells Splunk which stanza in transforms.conf to look at, and the REGEX entry in transforms.conf tells Splunk what text to match on. Be careful to to specify a regex pattern that will match only on the load balancer hits, so that you don't discard legitimate traffic entries. ... View more

The simplest way in that case would be to apply the filtering transform to a source instead of a sourcetype . Do the servers you wish to filter log to different filenames? ... View more

It's true that you can't filter the events out at the Universal Forwarder, but you can still do this. Filtering of this kind will occur at the point at which the data is indexed, so you have two main options. The filtering configuration is pretty much the same either way (routing to nullQueue); it just depends on where you configure it. a) Switch to a Heavy Forwarder Your filtering will be at the web server Uses more CPU and memory resources on your web server Will generate less network traffic (filters entries before sending) Filter entries go in props.conf / transforms.conf on the web servers b) Perform the filtering at the central indexer Lower resource usage on the web servers More network traffic (all logs are sent across the network) Somewhat simpler configuration on the web servers Configuration is done in a single place Filter entries go in props.conf / transforms.conf on the indexer Personally, I'd go with option (b), especially if you have multiple web servers and aren't using the deployment manager. A third option, sometimes good for remote sites, is to have a single heavy forwarder acting as a bridgehead, and have Universal Forwarder -> (Heavy Fowarder w/ filtering) -> Central Indexer. ... View more

If you're an Enterprise (i.e., paid) customer, then yes. See the Contact Us page here: [http://www.splunk.com/view/contact-us/SP-CAAAAH7][1] (Retired) New Link http://www.splunk.com/en_us/about-us/contact.html#tabs/customer-support ... View more

Something like sourcetype=XXX TAGS | head 1000 | dedup tag | sort tag should help with the dropdown search. Depending on what your data looks like, you may need to adjust for performance, and 1000 records may or may not be enough to see all tags. There are some things you can do if it's too slow (scheduled searches in advance or summary indexing), but basics first... ... View more

FWIW, the use of endswith in transactions is a little counterintuitive. endswith=(EventCode=600) wouldn't actually require that a transaction contain an event 600. It just means that if an event 600 is found, it will be the last event in the transaction. The real point of using endswith for something like this would be to make sure that an event 500 will be recognized as a re-infection. If you have a Detected, then Cleaned, then Detected, you want to make sure that you don't report that as clean. ... View more

Make sure that what you have in props.conf for [mysourcetype] matches the sourcetype field on your events exactly. Also try running a search for source=/mysource.log TAGS | kv extract-tags , and see if you get the mytags field. If it works that way, Splunk isn't running the extractions automatically, typically because props.conf is wrong. mytag won't show up until you can see mytags , since the process to get the individual tags relies on the value of the whole tag section being extracted first. It's ok if some events don't have any tags; the fields will just be missing for those. ... View more

Interestingly enough, Splunk 4.3 sets this value when creating new sourcetypes as part of the initial setup 'Add data' screens and choosing "Set a new sourcetype". Certainly that would seem to imply that the warning isn't warranted, but it would be nice to see confirmation and some insight into the original reason for the warning. ... View more

In the example XML, it first runs a search and fills the dropdown with the results. The search string is what goes in the 'what goes here' bit. After the search is run, it will fill the dropdown from the resulting values. You need to also replace the 'suser' from the example with 'mytag' or whatever you call your extracted field. Then, <choice> tells it to append a hard-coded option to the menu as well. IF you can be sure of knowing all possible tags ahead of time, just use the hardcoded option and put them in manually. Either way, you still need to extract the fields first. ... View more

Usually there are two ways. Either look at the value of linecount (I said search linecount>1 before, but yours may be more than 1 since you're dealing with multi-line events. Or int this case a better way is probably to pipe to search again and put something like | search NOT EventCode=500 at the end instead of the search on linecount. (The transaction should leave you with a multivalue field for EventCode ). There may also be a way to do it with streamstats and some clever search/where clauses -- would have to give that approach some more thought when I'm more awake. ... View more

Definitely transactions. Using the 'startswith' option may help, as may the linecount field. Something like: SourceName="AV Server" EventCode=500 OR EventCode=600 | transaction Computer,Message startswith=(EventCode=500) endswith=(EventCode=600) | search linecount>1 ... View more

1) Field Extraction transforms.conf: [extract-metric] REGEX=(\S+):(\S+) FORMAT=$1::$2 props.conf: [mysourcetype] REPORT-metric=extract-metric Or, if you just want search-time results, skip all this and use the extract command as shown below. 2) Timechart Reporting If your metrics field names follow a pattern, you may be able to use wildcards. For example if they all start with 'metric_', you can do: | timechart avg(metric_*) In a real pinch, you can use this (rather ugly) method: sourcetype=whatever ... | fields + _time,_raw | extract pairdelim=" " kvdelim=: | timechart avg(*) First, filter out all fields except _raw and _time . Then use extract to add back the fields containing your metrics. Now you can use timechart avg(*) , since only your metrics fields remain to be matched by the wildcard. ... View more