We are running a rails application and are using splunk to parse our rails logs. We have a search-head and 2 indexers. On the indexers, I have added the following to /opt/splunk/etc/apps/search/local/props.conf to ensure that the logging for each rails request is are parsed as a single event: [(?:::){0}*rails] LINE_BREAKER = ([\r\n]).* [\r\n]+Started (POST|GET) Each application server runs a forwarder. The rails log for each project is added using the following type of command, so that the source types match the pattern in the above stanza. /opt/splunkforwarder/bin/splunk add monitor -source '/home/builder/abitlucky/web/luckyonrails/log/production.log' -sourcetype project1-rails /opt/splunkforwarder/bin/splunk add monitor -source '/home/builder/abitlucky/web/luckyonrails/log/production.log' -sourcetype project2-rails The issue is that I am seeing every single rails request duplicated in the search application. If I click on 'Show Source' for both of the events for a duplicated request, I get strange results. For some cases, it highlights the original request in the source for one of the events and a completely different request in the source for the duplicate (i.e., the request it shows as the source does not match the request I clicked 'Show Source' for). In other cases, it highlights the original request in the source for one of the events, and then highlights a second listing of that request in the source for the second event. In these cases, I can see the request repeated in the source it is showing me. However, if I go back to the original log, the request only shows up once, so the repeated request in the source Splunk is showing me is a phantom/fake request that was not in the original log for these cases. I'm not sure if that makes sense without actually seeing what I am referring to, but I explained it as best I could. Has anyone seen this behavior? What could be causing it? ... View more

My set up is that I have splunk forwarders sending data to two load balanced indexers. I then have a search head that uses the indexers as search peers. I have a standard rails log file where every new request starts with a line like: Started (GET|POST) "/some-path"46 I want to extract "/some-path" into a field rails-action-path. I was using the documentation here to set up search time extraction using EXTRACT: http://www.splunk.com/base/Documentation/4.2.1/Knowledge/Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles The EXTRACT directive is in $SPLUNK_HOME/etc/apps/search/local/props.conf on my search-head as follows. [(?:::){0}*rails] EXTRACT-<rap> = Started \S+ (?<rails_action_path>[\S\/"\.]+) After restarting splunk on my search head, I can see the field in the field list under "Other interesting fields". All valid values of the field show up with relative percentages. However, if I click on one of those values to search by it, I get 0 results/No matching events found. Given that it just showed me the count of all the events with that value, that doesn't seem right. Note that if I search by field="*", I get all results, but any specific value returns no results. Can anyone help me figure out what's going wrong here? Thanks, Leigh ... View more

I am new to splunk so forgive my ignorance. My set up is that I have splunk forwarders sending data to two load balanced indexers. I then have a search head that uses the indexers as search peers. I am reading documentation about setting up search-time field extraction in props.conf. I have been playing around with it and it's not behaving as expected. However, I just realized, I'm not sure if I am supposed to be modifying props.conf on my search head or on my indexers. I was doing it on my search head with no success, but then it occurred to me that since the search head uses the indexers as search peers, maybe it should be done there? Can anyone confirm the correct place to put the field extractions? Thanks! ... View more

I refered to the following documentation to try and get this working: http://www.splunk.com/base/Documentation/3.0.2/Admin/CreateAdditionalSearchFields I have a standard rails log file where every new request starts with a line like: Started (GET|POST) "/some-path" I want to extract "/some-path" into a field rails-action-path. I added the following to $SPLUNK_HOME/etc/system/local/transforms.conf (note that the documentation says to add it to $SPLUNK_HOME/etc/bundles/local/transforms.conf, but there is no bundles directory in my installation and my other conf files have all been in the system directory). [get-rails-action-path] REGEX = Started \S ([\S\/"\.]+) FORMAT = $0 rails-action-path::$1 DEST_KEY = _meta My sourcetype stanza for the rails logs in $SPLUNK_HOME/etc/apps/search/local/props.conf is as follows (again note, my props.conf has always lived here although the documentation refers to $SPLUNK_HOME/etc/bundles/local/props.conf). [(?:::){0}*rails] LINE_BREAKER = ([\r\n]).* [\r\n]+Started (POST|GET) TRANSFORMS-rap = rails-action-path Note that the last line was the only part I just added. The wildcard source type for the rails logs and the LINE_BREAKER have been there for awhile and behave as I expect. After restarting my indexer with these changes, I do not see rails-action-path in the field list. Any ideas? ... View more

I assume there is no way to do what I want, but I figured I'd ask anyway. I have a background job processor that logs data about the jobs, as follows. [2011-06-13 12:10:31,DEBUG,ip-10-126-214-147(18959)] KtEventReport 82245248 completed after 0.1432 [2011-06-13 12:10:31,DEBUG,ip-10-126-214-147(18937)] KtEventReport 82245253 completed after 0.1443 [2011-06-13 12:10:31,INFO,ip-10-126-214-147(18959)] ,host:ip-10-126-214-147 pid:18959] 1 jobs processed at 5.9643 j/s, 0 failed ... [2011-06-13 12:10:31,INFO,ip-10-126-214-147(18937)] ,host:ip-10-126-214-147 pid:18937] 2 jobs processed at 6.2623 j/s, 0 failed ... [2011-06-13 12:10:31,DEBUG,ip-10-126-214-147(18944)] KtEventReport 82245256 completed after 0.1428 [2011-06-13 12:10:31,DEBUG,ip-10-126-214-147(18925)] KtUserReport 82245257 completed after 0.1455 [2011-06-13 12:10:31,INFO,ip-10-126-214-147(18944)] ,host:ip-10-126-214-147 pid:18944] 1 jobs processed at 6.5080 j/s, 0 failed ... [2011-06-13 12:10:37,DEBUG,ip-10-126-214-147(18952)] KtEventReport 82245275 completed after 0.1444 [2011-06-13 12:10:31,INFO,ip-10-126-214-147(18925)] ,host:ip-10-126-214-147 pid:18925] 1 jobs processed at 5.9259 j/s, 0 failed ... [2011-06-13 12:10:37,DEBUG,ip-10-126-214-147(18952)] KtEventReport 82245282 completed after 0.1435 [2011-06-13 12:10:37,INFO,ip-10-126-214-147(18952)] ,host:ip-10-126-214-147 pid:18952] 2 jobs processed at 6.4665 j/s, 0 failed .. Note that the number in parenthesis at the end of the timestamp is the process ID of the process handling the job. I would like to break up my events into batches run for each process ID. However, the various batches for each process ID are interleaved in the logs. If I understand correctly, you can only break up the logs into events serially, so what I want to do probably isn't possible. However, if I am wrong about that and there is a way to do this, let me know! To be clear, the above log would be broken into the following events. event 1: [2011-06-13 12:10:31,DEBUG,ip-10-126-214-147(18959)] KtEventReport 82245248 completed after 0.1432 [2011-06-13 12:10:31,INFO,ip-10-126-214-147(18959)] ,host:ip-10-126-214-147 pid:18959] 1 jobs processed at 5.9643 j/s, 0 failed ... event 2: [2011-06-13 12:10:31,DEBUG,ip-10-126-214-147(18937)] KtEventReport 82245253 completed after 0.1443 [2011-06-13 12:10:31,INFO,ip-10-126-214-147(18937)] ,host:ip-10-126-214-147 pid:18937] 2 jobs processed at 6.2623 j/s, 0 failed ... event 3: [2011-06-13 12:10:31,DEBUG,ip-10-126-214-147(18944)] KtEventReport 82245256 completed after 0.1428 [2011-06-13 12:10:31,INFO,ip-10-126-214-147(18944)] ,host:ip-10-126-214-147 pid:18944] 1 jobs processed at 6.5080 j/s, 0 failed ... event 4: [2011-06-13 12:10:31,DEBUG,ip-10-126-214-147(18925)] KtUserReport 82245257 completed after 0.1455 [2011-06-13 12:10:31,INFO,ip-10-126-214-147(18925)] ,host:ip-10-126-214-147 pid:18925] 1 jobs processed at 5.9259 j/s, 0 failed ... event 5: [2011-06-13 12:10:37,DEBUG,ip-10-126-214-147(18952)] KtEventReport 82245275 completed after 0.1444 [2011-06-13 12:10:37,DEBUG,ip-10-126-214-147(18952)] KtEventReport 82245282 completed after 0.1435 [2011-06-13 12:10:37,INFO,ip-10-126-214-147(18952)] ,host:ip-10-126-214-147 pid:18952] 2 jobs processed at 6.4665 j/s, 0 failed .. ... View more