You may find some of the transformer examples here helpful as well: http://splunk-base.splunk.com/answers/3470/consolidate-similarly-named-log-files-into-a-single-source/4420 ... View more

I suspect the issue is the order in which splunk is performing the lookup vs field extractions. Since you are extracting fields in your search explicitly, then you will want to use the lookup command explicitly after your fields have been extracted via the rex commands. Otherwise, I believe that splunk attempt to do automatic lookups immediately after your base search (which is simply source="dns2.log" , at which point in, extract_domain does not yet exist.) I would suggest you try: <your existing search> | lookup domainstatus domain OUTPUT status Once you get this working (assuming you want this to be done for other searches too) then I would suggest moving your field extractions to configuration files and then enable an automatic lookup for your source (or more preferably, for your specific sourcetype.) I'm struggling to find the exact search time operation order in the docs, or I'd link to it here. ... View more

Have you looked into using exporttool and importtool ? Another option is to simply move an entire bucket. (You have to be careful to use a bucket id that's not already in use. You may find this answer useful: http://splunk-base.splunk.com/answers/25174/how-to-exportimport-events-from-indexes ... View more

This is a hack to log the duration of requests in splunkweb. This allows you to (1) log the time of very long requests (any request over a specified number of milliseconds), and (2) log the request processing time for each request in the web_access.log access log. If you are not at all familiar with python, and/or don't feel comfortable poking around in the splunkweb (cherrypy) code, than you should avoid this hack. You have been warned. NOTE: (1.) You should backup all files before making any modifications. (2.) Be aware that any splunk upgrade will overwrite your changes. (3.) This is intended to be used to temporarily monitor request processing times, and hasn't been tested on a large scale or for long-term production use. Your millage may vary. (4.) Please read all the instructions before beginning. Okay, enough disclaimers. On to the fun stuff... Overview Basically, this hack involves the following: Install a simple CherryPy custom tool that captures the start/end times of your request. (Basically you just copy-n-paste into a file.) Update the core splunkweb startup script ( root.py ) to import the CherryPy custom request logging module. Add new entries to the web.conf to enable our custom cherrypy tool. If desired, modify the customlogmanager.py to add request processing time to each request in the splunkweb_access.log Steps Step 1: Create new 'requesttime' python module. Create a new file $SPLUNK_HOME/lib/python2.6/site-packages/requesttime.py , with the following contents: # The idea and code borrowed from here: # http://groups.google.com/group/cherrypy-users/browse_thread/thread/cb520cfa6588b550 import cherrypy, logging, time class RequestDurationTool(cherrypy.Tool): """Tool to provide logging if a request exceeds a configured duration (in ms) set by the threshold configuration parameter""" CFG_THRESHOLD="tools.request_duration.threshold" def __init__(self): cherrypy.Tool.__init__(self, 'before_request_body', self._start_request) self.logger = logging.getLogger('splunk') def _setup(self): cherrypy.Tool._setup(self) # Attach extra hook to compute timing after both raw processing and I/O cherrypy.request.hooks.attach('on_end_resource', self._end_resource) cherrypy.request.hooks.attach('on_end_request', self._end_request) def _start_request(self, **kwargs): cherrypy.request.request_start = time.time() def _end_resource(self): cherrypy.request.request_stop = time.time() def _end_request(self): # Elapsed time and threshold is in ms elapsed = round((time.time() - cherrypy.request.request_start) * 1000) limit = cherrypy.request.config.get(self.CFG_THRESHOLD, None) if (limit is not None and elapsed > limit): rsc_elapsed = round((cherrypy.request.request_stop - cherrypy.request.request_start) * 1000) self.logger.log(logging.WARNING, 'Request duration: %d [%d] ms (%s)' % (elapsed, rsc_elapsed, cherrypy.request.request_line)) cherrypy.tools.request_duration = RequestDurationTool() Step 2: Update your root.py main application The root CherryPy script must be updated to import our "requesttime" module. (Make a backup of the original root.py file before starting.) This file is located here: $SPLUNK_HOME/lib/python2.6/site-packages/splunk/appserver/mrsparkle/root.py Add the following python block to the top of the root.py script. When you are done, the first few lines should look like this: #!/usr/bin/python # CUSTOM HACK: Import the "requesttime" cherrypy tool try: import requesttime except Exception, e: print "Failed to import requesttime cherrypy request time tracker!!" print e # END OF CUSTOM HACK Step 3: Enable the request_duration tool in web.conf In $SPLUNK_HOME/etc/system/local/web.conf , add the following to the [settings] section: # Enable custom cherrypy toolbox for debugging request processing time # NOTE: Comment out these two lines if you experience problems starting splunkweb!! tools.request_duration.on = True tools.request_duration.threshold = 5000 The threshold line is optional. If you choose to perform step 4, and you do not want to log long requests separately, then you can exclude this entry. Note: After a splunk upgrade, the root.py script will be replaced and the above configuration entries could prevent splunkweb from starting. You will see these errors in web_service.log . You will need to redo step 2 after the upgrade, or comment out these entries in web.conf . Step 4: Enable request duration logging in web_access.log This step is optional, if you only want to log really long request, then you can skip it. If you want to do any trending, then you'll probably want to capture the timing of all requests. We will need to update the following python module: $SPLUNK_HOME/lib/python2.6/site-packages/splunk/appserver/mrsparkle/lib/customlogmanager.py Towards the bottom of this module, you'll find this line: self.access_log.log(logging.INFO, self.access_log_format % atoms + ' - ' + get_request_id()) Replace that line with the following block of code: try: timming = "%dms" % (round((cherrypy.request.request_stop - cherrypy.request.request_start) * 1000)) except: timming = "-" self.access_log.log(logging.INFO, self.access_log_format % atoms + ' - ' + get_request_id() + ' ' + timming) You will need to make sure you have the indentation correct, or the python module will have issues; this could prevent splunkweb from even starting. You may also want to setup field extractions for the request duration, which can be done by adding the following to your $SPLUNK_HOME/etc/system/local/props.conf config file: [splunk_web_access] EXTRACT-request-time = \s(?<req_duration>\d+)ms$ ... View more

You may also want to take a look a the following question/answer: Consolidate similarly named log files into a single source If you want to consolidate all files in a directory as a single source then using the [monitor] stanza that has an explicit source= setting will work, but if you have different types of log files in a single directory, then I recommend using a transformer-based approach (as detailed in the link above). ... View more

Great question. +1 Why has no one else voted this up? ... View more

Following up on what Genti said. You really don't want do indexed fields, you want search-time field extractions. So move your "proxy" from a TRANSFORMS to a REPORT entry. Also, I'd be extra careful extracting a field called "_time", since this is a built in and very important field used internally by splunk. ... View more

So, is you issues with approach 1 simply that you can't determine if the transaction is still open or not? If so there may be a very simple solution. You could determine if the transaction is complete by simply tacking a search or eval to the end. | file /tmp/tnsexp.log | extract | sort - _time | transam tnsid altid | eval is_complete=searchmatch("term") It is much better for performance if you can specify startswith or endswith because transaction can be evicted once they are complete (uses less memory); but it may be possible to get away without them using appropriate maxspan and/or maxpause settings. I have some more complex approaches that might work here; but they involve subsearches and/or nested transactions, so there's no point going there if the above gets the job done. ... View more

Yes, this is possible. I would suggest that you provide an example of your two different searches. (Use the "edit" link, to update your question) ... View more

Start by removing your [tcp://:9997] stanza, you shouldn't have both a splunktcp and tcp listener on the same port like this. ... View more

Wow, that's pretty intense. Looks like this may be a good candidate for a macro; I'd hate to have to retype that several times. 😉 Nice work. ... View more

It's the same for both. ... View more

See the following docs page: http://www.splunk.com/base/Documentation/latest/Admin/ConfigureSplunktostartatboottime BTW, I do like to make a minor change to Splunk's default boot script that makes the startup work in a few extra situations. (For example, after a splunk upgrade) By default, you'll see a line that looks like this (your's may be slightly different depending on what user splunkd is running as on your system, on mine the user is simply called "splunk"). This is in the splunk_start() function. I change this line: su splunk -c "\"$SPLUNK_HOME/bin/splunk\" start" to the following: su splunk -c "\"$SPLUNK_HOME/bin/splunk\" start --accept-license --answer-yes" This ensures that any migration questions or license acceptance issues do not prevent splunk from starting up on an unattended boot. There are other startup options as well, but these are the two that I've found to work best for us. ... View more

bshuford, please post your full search in your original question instead of putting it in a comment for readability. (Use the "edit" link under your question at the top.) Also, you could try adding "field" to your rex command, like so: rex field=Member_ID "CN=(?<out>[^,]+), but that shouldn't be necessary unless "CN=" appears multiple times in your event. ... View more

Ron, I agree that getting the permissions all setup properly can be an issue, and it's often difficult to find which piece is missing. But I too have seen some situation where it appears that the only "solution" to getting a lookup working properly, is to restart splunkd like jambajuice is asking about. There does seems to be something glitchy about this, but I haven't taken the time to track it down precisely. ... View more

Another approach, is to look at who is currently authenticated to your splunkd process. This isn't a really a search, but it may give you the info you are looking for. https://your-splunk-server:8089/services/authentication/httpauth-tokens Note the "userName" field. ... View more

cadeli, please update your original question (use the "edit" link) to include additional examples of what your event looks like and what phone numbers you are looking to extract. Be sure to include examples of different types of events that you would like to extract phone #s from (it sounds like you may have multiple formats, based on the fact that your question and your above comment show two different message formats.) ... View more