I've been having trouble getting a host override transformation in my props.conf/transforms.conf to work and want to figure out if the problem is in the config files or in the running environment. I've got a centralized log server (forwarder) that writes out logs with data drawn from multiple hosts. We want to override the default host so that the indexer receiving the logs shows the original host name, not the log server's name. As I understand it: Performing a host override is possible and normal. The common practice is to run the transform on the indexer. For internal policy reasons, this is either impossible or at least not likely. Only a 'heavy' forwarder will process the transform, not a light or universal forwarder. What I'm trying to confirm now: What kind of forwarder is running on my centralized log server? Is there a way to get some feedback from the forwarder regarding the application of transforms? splunkd.log doesn't seem to have any relevant entries. In other words, how can I check that the forwarder is a heavy forwarder and how do I check what (if any) transformations are being applied? For info: splunk enable app SplunkForwarder The SplunkForwarder is listed as unconfigured, enabled, and invisible. I've got config files in place and the Web management GUI lists the forwarding rule and says that it's enabled. That doesn't add up to a coherent picture for me. Is this what a heavy forwarder should look like? splunk list-forward-server This returns "Active forwards: None" but "Configured but inactive forwards" lists the forward that I'm using. Events are forwarding without any obvious problem. I think that the SSL part of the forward may not be working correctly, if that's relevant. Other info: * Events are flowing through to the Splunk indexer. So, forwarding is happening. I'm using custom logs with custom types defined by the Splunk forwarder. These types are recognized by the indexer so I take it that my props.conf and inputs.conf are in the right locations. ... View more

I'm having some trouble overriding the default host assignment and am hoping for some help. I've tested out a regex with rex that looks to work correctly but it's not seeming to work from props.conf + transforms.conf. For background, here's our setup: We've got a lot of custom logs. Each log has its original data stored in a SQL database. A single program on a single machine reads the rows off the SQL database and writes out the individual logs locally. Call this machine LogMachine. LogMachine has a copy of Splunk acting as a forwarder that sends all of the data into a remote Splunk indexer. So, all of the logs are written locally to disk on LogMachine and then forwarded on. For policy reasons outside of my control, this part of the setup is fixed. Namely, that we've got all of the logs on one machine and forward them from there. The problem is that every event has a host of LogMachine rather than the original host machine. I've looked at the docs and several threads here and it looks like I'm meant to define a transform and declare it in props.conf. I'm only testing out one log right now as I'm trying to get the mechanics sorted out before I work on doing the same thing for our other custom logs. Here's a snippet from a custom 'action' log: [23/Jun/2012:01:50:06 +0000] add appuser FANGLET-AU000-0000000000 0 1336643d3082237d75191d4362fbd941 - 1.0 - 345101-VM3 SRST [23/Jun/2012:01:51:38 +0000] add appuser FANGLET-US000-0000000000 0 9fb0638027e115dc36a313700ada3f54 - 1.1.4 - 345101-VM3 SRST [23/Jun/2012:01:51:53 +0000] add appuser FANGLET-AU-EGGPLNT 0 d1128ee5236b17a41825832b890a8091 cdma_spyder 1.0 10 345101-VM3 SRST [23/Jun/2012:01:52:47 +0000] add appuser FANGLET-AU000-0000000000 0 5d3ded5a3efbc9102c85e319d08c461d - 1.0 - 345101-VM3 SRST [23/Jun/2012:06:48:04 +0000] add appuser FRINDO-UK-EGGPLNT 0 c9e9d9c86fe1592e3427592c4c4bc6a buzz 1.0 8 345999-VM4 SRST [23/Jun/2012:06:48:20 +0000] add appuser FANGLET-AU000-0000000000 0 d0e3cc86221875df6485f28e6246bcf8 - 1.0 - 345999-VM4 SRST [23/Jun/2012:06:48:56 +0000] add appuser FRINDO-US000-0000000000 0 459d7c547c40efa025feb0ea9fd93998 - 1.1.4 - 345999-VM4 SRST [23/Jun/2012:06:48:57 +0000] add appuser FRINDO-US000-0000000000 0 8321965193395108fe7d85878f8c9a43 - 1.1.4 - SRST-Remote-2 SRST Here's the transforms.conf text stanza that applies: # transforms.conf [action_host_override] REGEX = (?i)^(?:[^ ]* ){10}(?P<host>[^ ]+) FORMAT = host::$1 DEST_KEY = MetaData:Host Here are props.conf stanzas that apply: # props.conf [source::.../action_log.txt] sourcetype = action [source::.../action_log.txt] TRANSFORMS-action-host=action_host_override I don't know if that needs to be in two stanzas or one - I've tried both with no change either way. Namely, the sourcetype is applied and the transform is ignored. For good measure, here's the inputs.conf stanza for this sourcetype: [monitor://C:\Program Files\SRTS\logs\action_log.txt] disabled = false index = srst followTail = 0 sourcetype = action I've tested that regex out with rex and it extracts the values that I expect. Namely 345101-VM3, 345999-VM4, SRST-Remote-2 and so on. Here's a rex sample set in the UI to the past 15 minutes: sourcetype="action" | rex field=_raw "(?i)^(?:[^ ]* ){10}(?P<host>[^ ]+)" I'm guessing that I need to change the regex for transforms.conf (?) I originally had a SplunkUniversalForwarder running but gather from an earlier thread that I need at least a full forwarder. I installed Splunk 4.2.5 and licensed it as a forwarder. Here's the machine setup: Win 32 Splunk 4.2.5 acting as a forwarder to an indexer running Splunk 4.2.2. I do not control the indexing machine and the person managing it is always short on time. So, I'd prefer to run the transformation before forwarding. I'd be extremely grateful for help resolving this problem! Thanks, ... View more

I've got a 4.2.5 server and may be migrating to a 4.2.2 server. I've looked at the docs and they say that there's no way to downgrade but that you can install an earlier version. Does anyone know if it is safe to install 4.2.2 on top of our 4.2.5 data? I'd need to change the server and about eight forwarders. Thanks in advance for any help ... View more

We've currently got a stand-alone 64-bit linux box handling inputs from a collection of Splunk forwarders, all on 4.2.5. $SPLUNK_HOME is still set to it's default and all of the data is flowing into the original, default index. We've got about 13 millions events, a lot of saved searches and dashboards. We're now looking at how to migrate our system over to an existing 64-bit Linux Splunk setup that's running on 4.2.2. I've been reading the docs and Splunkbase about migration but am still not clear on how this would actually work. I can zip/tar the original data, but how will that work on the new server? I guess what I'm after is that we've got a home directory on the new server with all of our old data, field extractions, etc. that doesn't interfere with the rest of the server. Is that possible, given that the data is currently in the standard home default? The server admin is happy to set up a specific index for us on the server farm, but I'm hoping to migrate our past events without export-to-CSV-and-reimport and to keep our existing searches, dashboards, etc. Thanks for any guidance or links. ... View more

I've got a series of events with a timestamp and two numbers, like so: "2011-05-29 22:54:06",68,31 "2011-08-15 10:20:33",143,76 "2011-09-15 10:56:09",63,27 "2011-09-20 20:32:15",0,0 "2011-08-20 09:23:19",0,3 The two numbers represent "success" and "failure" counts for a specific event. What I'd like to be able to sort out are stats for each of the numeric series as well as the ratio between success/failure over time. Average, min, max, stddev and counts over time for "success". Average, min, max, stddev and counts over time for "failure" Success/failure ratio over time. I've been banging away on this for some time, but I don't seem to be able to extract two numeric series from the same sequence of events. Am I running into a known limit? http://docs.splunk.com/Documentation/Splunk/4.3/User/ReportOfMultipleDataSeries If there's no way to do what I'm after with the data in the current format, would I be better off restructuing the data to make it easier to work with using eval()? "2011-05-29 22:54:06",success,68 "2011-08-15 10:20:33",success,143 "2011-09-15 10:56:09",success,63 "2011-09-20 20:32:15",success,0 "2011-08-20 09:23:19",success,0 "2011-05-29 22:54:06",failure,31 "2011-08-15 10:20:33",failure,76 "2011-09-15 10:56:09",failure,27 "2011-09-20 20:32:15",failure,0 "2011-08-20 09:23:19",failure,3 The example above is a simple case - two fields with numbers I'd like to trend and compare - and is just a starting point. I have more complex requirements but have to get the basics down before tackling anything harder. Thanks for any help or suggestions. ... View more

I've got a search where a heatmap really helps highlight values of importance. Unfortunately, the default colors tend to make the 'hotest' values very hard to read. Is there a way to customize the heatmap colors? I've check the docs and SplunkBase for information on tweaking the colors without luck. I've searched under 'heatmap' and 'SimpleResultsTable'. Thanks for any pointers. ... View more

I've got a search like this against a collection of Web logs: sourcetype="access_common" | ctable uri_path host The result is a ctable with URLs down the left, a count column for each of our servers and a grand total column on the right. How can you format the numbers that are in the cdata cells? (I've been having similar problems with summarizing over table results and xyseries results.) This fieldFormat statement does correctly format the grand total column: sourcetype="access_common" | ctable uri_path host | fieldFormat TOTAL=tostring(TOTAL,"commas") So far, I haven't figured out how to address the columns for each specific host. If I use the column name that Splunk produces, no error is generated but there's also no data. I'm guessing that the problem is that I don't know what the correct identifier for the column(s) that I'd like to format. Within the result data, I can see that there are definitions with "tag" elements for the various columns - but these ID values (c='6474' or c='327', etc.) are not stable. Each time you run the search, the IDs are generated from scratch. Is there a way to address the generated columns in a ctable? For that matter, is there something that works with other, related sorts of extractions like xyseries? Thanks a lot. ... View more