Just some thoughts. This is not really an "answer", but more than can will fit into a comment. How are you testing this, are you just letting splunk pickup new content after a restart? If so, let me just point out that you can do immediate testing with the following command: splunk test sourcetype /var/log/apache/my_log_file This will spit out all the props settings applied to this file. (Finding this utility has saved me countless hours of messing around; so I try to advertise it as much as possible.) Also, there are times where I've had to go into the $SLUNK_HOME/etc/apps/learned/local/sourcetypes.conf and do some housekeeping. I believe this is because once splunk identifies a sourcetype of a file, it prefers to stick with that same sourcetype unless somethings changes, and changing a rule entry may not be a strong enough suggestion. Normally this is the behavior that you want, but in this case you may find that deleting a few entries related to your file in question, may be helpful. Also note that the utility I mentioned will possibly create or update an entry in the learned folder as well. Additional thought: Sometimes its helpful to come at this from a slightly different perspective. I had a similar issue with my own custom apache longing sourcetype ( vhost_access_combined ) that was getting confused with access_combined , I think. The issue came down to the rule for access_combined was slightly too lose. So instead of just trying to make a better rule for my vhost_access_combined sourcetype, I found that I had to instead add an additional rule to the builtin access_combined to make it more restrictive in the first place. In my situation, I knew that my sourcetype would start with a named virtual host whereas access_combined starts with the clientip address, so I could add a new rule to access_combined to make sure that it starts with an IP address and therefore my own logs could then match against my sourcetype: [rule::access_combined] MORE_THAN_66 = ^\d+\.\d+\.\d+\.\d+[ ] Also note that I'm using MORE_THAN_66 and not MORE_THAN_75 which is used base config. We don't want to replace splunk's builtin definition, only refine it. In my case, I simply added this to my own custom config file. As always, make sure you are making changes like this in a local folder somewhere, I am not suggesting that you modify the entry in $SPLUNK_HOME/etc/system/default/props.conf . Again, I'm not proposing this as a solution to your problem, but as a different approach to consider. I should also point out that while this approach worked for me, I ultimately don't rely on it. I found that I ran into issues when my log files were rotated. The new log was too small for splunk to identify (I think it needs 100 or more line before it attempts rule-based recognition) and our apache volume is't super high, so we didn't have enough events and the result was that the sourcetype was wrong half the time anyways. Perhaps this has been improved or I was doing something wrong, but in any case, I went back to using a simple [source:/var/log/apache/vhost_access.log] stanza with sourcetype=vhost_access_combined setup on the forwarder on the apache server, and it solve all these issues quite nicely. Hope this gives you something to think about. ... View more

I think the biggest thing you have to decide how up-to-date you need your log data to be. Here are some options starting from the most "live" to the most infrequent. Forwarder: If using a splunk forwarder your log files are monitored and forwarder very quickly and normally from withing a search you see your events in under a minute after they occurred. That is ideal, but it sounds like this isn't an option for you. Also, I'm not sure what the overall performance is like when forwarding over the Internet (everything we have is intranet, so I can't speak to that. Although I assume enabling compression and encryption would be very helpful options here.) Synchronization: Push files from hosted environment: Another option is to synchronize your files to your local network. You didn't mention what platform you are on, but since you mentioned cron , I would assume it's unix based. So if you can use rsync to push files to your local system (or preferably, rsync via ssh ) you could setup a job to push your log files to you incrementally, and then your log files could be indexed incrementally, as often as your setup your cron job to run. (You would probably want to play around with rsync's settings on how it handles temporary files. It may be preferable to have your log files appended, instead of copied to a temporary file which is then updated and renamed over-top of the original file. I image splunk would do well with any of these, but there is probably some options that will work better than others.) Do a recursive web copy using wget (Thanks to Jrodman for this idea). You could use wget or curl to recursively pull your web log files over http or ftp. If you combine this with a "continue" mode, then the logs could be pull incrementally. Pull entire log files You could setup a daily (or weekly?) job to pull the log files of of FTP and put them in a local location that is monitored by splunk. You would want to do this after your log file rotation so that you don't have to worry about event being written to it after you copy the file. You may also be able to gzip the file on the hosted environment before you pick it up, not sure what flexibility you have here. If you only want your data indexed once a day, than this would work fine. Once the file is copied locally, splunk provides a much of different ways to have it loaded. You could use the de facto "monitor", or if you want the local copies deleted after they are indexed, you could use a "batch" input. And then you have various one-shot input options too. In fact I think you can even upload a log file via an HTTP POST if you really want to (think this is limited to 500Mb, but I could be wrong, this probably would be the best option to start with anyways.) Here is a quick example of doing a recursive copy using wget (option 2.1). This will copy logs to directory structure under C:\RemoteLogs . wget --user=USER --password=PASS --continue --recursive --level=3 --no-parent http://my.site.example.net/path/to/my/logs -O C:\RemoteLogs\ NOTES: This assumes you've installed a wget client for windows. As with all HTTP traffic, your content will be transfered in an unencrypted form, so if your data is sensitive this may not be an acceptable solution. Also be aware that I haven't actually tested this, and some of the options are purely a guess, so be prepared to tweak the example. To run this on a regular basis, you should be able to set this up as batch file and set it up as a scheduled task. For disk space management purposes, this approach will require some sort or log file cleanup process(es) that firstly, removes old log files from the hosted environment, and then secondarily, removes the local copy of the log files. ... View more

This is technically more of a comment than an answer (both jrodman and gkanapathy covered the topic well.) But comments have limited formatting, so I'm posting it here.. BTW, it looks like your regex got scrambled when you posted your question. With that said, you may want to consider the following regex tweaks: Drop the trailing un-named group which probably isn't necessary. (Or give it name if you want it.) Using non-greedy dot matching is normally a good thing (when compared to simple .* ), but this can still be costly because of back-tracking within the regex-engine. Perhaps it would be more efficient to say "stop matching one you find a "]". As long as you don't have nested square braces, you should be able to match your values using [^\]]+ rather than .*? . This should work well for you: EXTRACT-dbflds = ^[^\]]+\] \[(?<locationid>[^\]]+)\] \[(?<hostname>[^\]]+)\] \[(?<database>[^\]]+)\] \[(?<instance>[^\]]+)\] \[(?<pid>[^\]]+)\] \[(?<thread>[^\]]+)\] Just some thoughts. (Hopefully this will actually be formatted correctly when it gets posted...) ... View more

Have you considered using lookups instead? Lookups certainly do not replace tags, but they do functionally overlap in certain situations and you may find that lookup are are more natural solution. (You haven't given any details as to what you are trying to do, so lookups may or may not be a good option for you.) I know there are a number of places where we used tags in Splunk 3.x (because that was the best option then) and we are now moving to using lookups in Splunk 4.0+. I'm guessing that you were using tagcreate from a saved search in Splunk 3.x? If so, you may be able to use lookups in 4.0. In Splunk 4.0+, you can use saved searches to generate lookup tables: Take a look at the action.populate_lookup.* settings in the savedsearches.conf reference. If you want to append your lookup table instead of replace it each time your saved search runs, you should be able to leverage the inputlookup command's append=t option. For example, if your saved search creates a lookup table named my_lookup.csv , then you should be able to use a search pattern like this in your saved search: <<base search>> | fields + key val1 val2 | inputlookup append=t my_lookup.csv | dedup key Lookups also have the advantage of being able to match against more than one field, and can can return more than one value as well as adding a simple date-effectivity into the mix. Whereas tags tie only to a single field, and can't represent changes over time. ... View more

You will want to setup custom sourcetypes for these log files. Then setup your event breakers and optionally timestamp parsing logic. This is a big topic, and the docs are the best place to get started. http://docs.splunk.com/Documentation/Splunk/5.0/Data/WhatSplunkcanmonitor You may also want to getting familiar with the props.conf file. http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Propsconf If you update your post to be more readable (with <pre> tags), then I'm sure someone will take a crack at getting you an example props entry to get you started. It can be a little overwhelming at first, but once you get familiar with setting up these kinds of sources, it's pretty easy to get splunk to index almost any input. Update: Try something like: [my_wacky_log_file] SHOULD_LINEMERGE = True BREAK_ONLY_BEFORE_DATE = True # Breaking events on lines like: "ERR: Thu Jan 28 05:15:29 2010" TIME_PREFIX = ^\w+:\s+ TIME_FORMAT = %a %b %d %T %Y Additional note: If you log content is truly difficult to get splunk to properly handle out of the box, you always have the option of reading a log file with a custom input script which does some pre-processing work on your log file. (For example, if you need to event breaking when the prefixed number changes (In your example, I see 5688: and 4436: ), that would be pretty easy to do programmatically, but not something splunk would do out of the box. At least, not at index time.) You could even translate your timestamps to a common format, or make different parts of your log file appear to be coming from different sources (filenames), It's very flexible. Helpful docs links: Scripted inputs overview Assign default fields dynamically I'd recommend keeping the scripted input option in reserve, I've rarely found it necessary to take this approach, but it's good to know that you have this option when you truly need it. ... View more

I don't think your example is "delimiter based" exactly. I think the best approach for this would be to add a transforms.conf entry for this. (You can often add regex-based extractions more quickly using EXTRACT- as of Splunk 4.0+, but unfortunately this is one case where you have to do make a full-blown transforms.conf stanza because we need repeating matches, which EXTRACT doesn't support.) Also, the rex search command does not work because you are extracting both the field name and the field value; so using a transforms entry is really your best option. Something like this should do the trick: [my_kv_pairs_with_spaces] REGEX = \[([^=]*)=([^\]]*)\] FORMAT = $1::$2 MV_ADD = true Then make sure your add an entry in your props.conf file that points to your transformer. For example: [my_sourcetype] REPORTS-my_fields = my_kv_pairs_with_spaces Note: I'm assuming since you trying to use the kv command that splunk isn't just extracting these correctly out of the box for you. If you haven't tried letting Splunk do the extractions automatically, I would definitely start there. ... View more

You could filter out events based on the _time field using a where search command. This is more difficult to setup that using date_wday , but it's very flexible. The basic approach would be to preform your search, then grab the timerange of your searchs using the addinfo search command. Then use the where search to filter out the unwanted events in the middle of your search range. For example, say you were searching over a 1 hour window, but want to remove the center 30 minutes (so remove events between 15 after to 45 minutes after), you could do a search like this: <your search> earliest=-1h@h latest=@h | addinfo | where _time < (info_min_time+900) OR _time > (info_max_time-900) Some additional thoughts: Along with info_min_time , info_max_time which I used in this example, there is also info_search_time which could be used if you wanted to do some time operations relative to the system time when you run your search. Note that you can also use _indextime here, if you wanted to look at when your events were actually indexed rather than when your events occurred; which is sometimes interesting to look at. ... View more

In order to do event association like this, you must make sure that (1) you also have a matching closing event in the normal case, and (2) that you can uniquely associate the opening and closing events, which in your example should be possible using the pid ( 9125 ), and optionally the user ( test ) as well. You should be able to use a search like this to group your events into transactions: pam_unix sshd (closed OR opened) | transaction fields="host,pid" maxspan=30m startswith=opened endswith=closed keepevicted=t If that works, you can add | search closed_txn=0 to the end of your search to filter out the incomplete events. This works because complete transactions mean you have both an opened and closed event, which is not true if the session took more than 30 minutes. You can also play around with extracting the user field (if you haven't done so already). I'm guessing that this is something you will want if your goal is to report on who is logged in past your 30 minute window. Something like | rex "user (?<user>\w+)" should do the trick, but I haven't tested it. You can add this to the list of fields if you want as an extra safety, just in case your pids are reused very frequently, for example. So your final search would look something like: pam_unix sshd (closed OR opened) | rex "user (?<user>\w+)" | transaction fields="host,pid,user" maxspan=30m startswith=opened endswith=closed keepevicted=t | search closed_txn=0 You may also find the duration field added by the transaction command helpful. You could use this to chart average durations by user, for example. ... View more

Sometimes renaming _time can have unwanted side effects. So it's worth noting that you can always make a copy of the _time field and then manipulate the copy (this many not work for your exact example, but sometimes this can be a better approach.) So if you wanted to make a field called my_time and wanted it formatted as just YYYY-MM-DD , then you could do something like this: | eval my_time=_time | convert timeformat="%Y-%m-%d" ctime(my_time) Also keep in mind that there are other ways to control chart formatting, especially in Splunk 4.1. But that seems like that may be a different question than what you are asking here. (You didn't mention which version of Splunk you are running. The timeformat piece may be new in Splunk 4, I don't remember) ... View more

For the simplicity, let me consolidate the answers and comments here: NOTICE: Splunk 4.1.1 and later add a migration note about these permissions changes when during the upgrade. So hopefully fewer users will run into this issue. The migration notes point your to the following docs page. In Splunk 4.1. it is now possible to set permissions on individual lookup tables (aka .csv files) in the lookups folder of an app. Because of this new control in Splunk 4.1, you have the option to grant or restrict access to these files. There are actually 3 different sets of permissions to control access to the lookups now. You basically have two options: (1) Disable your lookups to make the error message go away, and use the lookup search operator instead. Or (2), make your transforms and lookups global (or otherwise tweak your permissions till you get it all working again...) Option 1: Disabling your automatic lookup Assuming your have sensitive info in your lookup file, you can avoid this error message by disabling (comment-out) or changing the permissions of your LOOKUP-<class> entry in your props.conf file. You can do this from the UI via Manager » Lookups » Automatic lookups, then find your stanza name (which will be based on your source, sourcetype, or host and the name of your LOOKUP line) and change the permission using the Permissions link. If you prefer editing config files directly, you can also set your permissions in metadata/local.meta with an entry like this: [props/my_sourcetype/LOOKUP-MyFieldLookup] access = read : [ * ], write : [ admin ] export = none Keep in mind, that if you choose to completely remove (or comment out) your LOOKUP entries, rather than change the permissions, then you will need to use the lookup search command now, since the lookup fields will no longer be "automatic". Option 2: Enable global lookup access You can go back to the globally-available behavior of Splunk 4.0 by setting your lookups permissions using the UI (Manger >> Lookups >> Lookup Table files, then click on the Permissions link). This can also be done by updating the metadata/local.meta file manually. To allow all access to all csv files from any application, add an entry like so: [lookups] export = system Or, for a single lookup file: [lookups/my_lookup_file.csv] export = system NOTE: You can also specify permission to the Lookup definition (based on the transforms.conf name), in same way as in Splunk 4.0. So if you want your lookup to be global, you would use something like: [tranforms/MyFieldLookup] export=system Again, this can also be done from the UI. In Splunk 4.1, navigate to Manger >> Lookups >> Lookup definitions, then click the Permissions link. ... View more