Frozen data is not available for searching, so think of frozen buckets as an "offline" bucket archive. The bucket data can be safely moved as it should never actually be in use, i.e. when buckets get moved to frozen directory, search can no longer see them. Based on your above configs, if you plan to leave the directories there in F you can move the data elsewhere, or you can change the coldToFrozenDir to a new location which has space and reload the config/restart splunk on your indexer to make sure it gets picked up. More info on coldToFrozenDir is available here: http://docs.splunk.com/Documentation/Splunk/6.2.1/Indexer/Automatearchiving ... View more

Also note, that if you use a full splunk instance as your forwarders here, you will not incur license cost unless these systems were configured to index the same data that you have them forward. You can also disable splunk web on heavy forwarders through the settings page or in web.conf with the following if you decide that you need the capabilities of a heavy forwarder: [settings] startwebserver = 0 ... View more

Right, you might want to reassess when you start ramping up your forwarding. 12 cores for a UF is pretty huge. At least with VMs you can safely oversubscribe if they are not actually using them, but there is no sense in total overkill. ... View more

Yes, you would want to at a minimum have a full splunk instance set up on a server meeting the system requirements. This would be your indexer and search-head which you would configure to receive data on the default TCP port or a port of your choosing. Then you would install the universal forwarder on your tomcat servers, configure them to output data to your indexer's hostname/IP, and port as well as configure to monitor the files you want to start indexing. This is pretty straight-forward stuff but it is not trivial to explain and documentation is already written to cover the process. See the following links as well as the rest of the Getting Data In and introductory section: http://docs.splunk.com/Documentation/Splunk/6.2.1/Data/Configureyourinputs http://docs.splunk.com/Documentation/Splunk/6.2.1/Forwarding/Introducingtheuniversalforwarder ... View more

Considering the dual-cores are recommended by splunk for UF and "light" forwarder, I would think that the app is can fully utilize multiple CPUs/Cores. On linux top and ps show multiple threads if you tell the tool to show threads. If you are not seeing the the throughput/utilization you expect, perhaps there is some other bottleneck. The first place I would look on a UF is in limits.conf, especially at "thruput" since the default on a UF is 256kbps if I recall correctly. ... View more

Yes, it can be done. Check out the discussion and documentation regarding using datetime.xml to do this: http://answers.splunk.com/answers/12015/setting-date-on-event-based-on-filename.html (there is some bad formatting on the config code but you should get the idea) http://docs.splunk.com/Documentation/Splunk/6.2.1/Data/Configuretimestamprecognition ... View more

Failing the above, you could also try: HEADER_FIELD_LINE_NUMBER = 25 The use of either option assumes that your application will not be appending headers to the same log if it is restarted and that the number of header lines remains consistent. If this is not the case then you will need to come up with a regex which matches all the header lines ( I am thinking of matching on not starting with a time string) and use a transform to send matching lines to the null queue. ... View more

Your question is not very clear to me but I will tell you that configuration items in splunk are spread in several directories depending on changes made at the app level and any system level changes and are merged according to precedence rules. You can read about how this works here: http://docs.splunk.com/Documentation/Splunk/6.2.1/Admin/Wheretofindtheconfigurationfiles You can use btool to investigate what the active configuration on a splunk instance: http://docs.splunk.com/Documentation/Splunk/6.2.1/Troubleshooting/Usebtooltotroubleshootconfigurations To capture every configuration item in a single file you would basically need to archive [SPLUNKHOME]/etc/apps and [SPLUNKHOME]/etc/system directories. ... View more

The original question has nothing to do with key/value equal-sign extractions. ... View more

try adding the reference to the correct extraction to the syslog sourcetype, if you have other types of data coming in as syslog, it might be impacted. The correct way to address this either requires breaking out different sourcetypes from your syslog data or doing something more advanced using an event based override as described here: http://docs.splunk.com/Documentation/Splunk/6.2.1/Data/Advancedsourcetypeoverrides If you have only apache data here you may be able to add this to the syslog sourcetype stanza in props.conf and have it work, but this may break not properly transform other events: REPORT-access = access-extractions This is what actually tells it what extraction definition to use. ... View more

What sourcetype is the data getting indexed as. The sourcetype on this input might be set to something other than access_common. IIRC splunk determines pretrained sourcetypes based on some of the first data in the input. So you may need to set the sourcetype of the input to access_common. ... View more

The first parts of each line in these events look like syslog data so this data is likely getting seen as a syslog sourcetype. The client IP field above is where it actually starts looking like combined apache access data. Events consisting of mish-mosh of two different sourcetypes is obviously not going to work with the built-ins so you either need to remove the part of the events that are not part of the pretrained apache access log sourcetype before input or implement a transform that trims all that syslog stuff before the clientip. Another way would be to customize either of the two extraction transforms to perhaps use bits from the other at which point you will have created your own syslog-httpd-access sourcetype. I wanted to share a little background on why it is not working but instead of doing all the work yourself, you might want to look at this: http://wiki.splunk.com/Community:StripSyslog ... View more

That will work too 🙂 ... View more

Hello, Do you have the TranReferenceLookup object defined as a database lookup with the fields ID and Name set up as your lookup fields? Also, what has worked for me is to set up advanced database lookup settings and specify the input field and query. For example: Input fields: TransID Query: SELECT Name FROM tablename WHERE ID = $TransID$ ... View more

It looks like your universal forwarders are trying to open connections to send data to your indexer and are having the TCP connection time out due to some sort of threshold on your gateway device. You could possibly rule out this being anything specific to splunk by trying other outbound tcp connections from the servers which would traverse the same gateway device during the peak loads. The comment about "reaching the limit of threads to the server farms" is to me ambiguous in f5 speak, so I would recommend that more detailed stats be collected about how many connections are in the various connection tables on the f5 when this is happening and get some of its load statistics as well. If the UF can't connect to the indexer, then the default or configured queuing behaviors will be in effect on the forwarder and depending on what is configured you may or may not be protected from data loss. See here for more on that: http://docs.splunk.com/Documentation/Splunk/6.2.1/Forwarding/Protectagainstlossofin-flightdata The behavior I observe on my universal forwarders is that TCP connections for forwarding data are not held open indefinitely so you will normally see one in ESTABLISHED state and a few in TIME_WAIT when looking at netstat on the forwarder. A few such connections is expected under load or even idle conditions in my experience. You might want to look at netstat on the forwarders and indexers during this time to get a rough idea of how many TCP connections on port 9997 are in your splunk systems connection table. I would expect that during the problems with your gateway device (the f5) you would see a connection or two in SYN_SENT state as they are waiting for the ACK from the server. This also confirms that there is a connectivity issue. You might want to consider using some type of alternate route if that is available if you are seeing some limitation on the f5 which cannot be overcome. In our environment we use SNAT on the f5s so that only the load-balanced application traffic flows through the f5 and the rest flows through a firewall. ... View more

A few questions: Are your servers using the f5 as their gateway back to the indexer? Have you looked at the load, connection count, bandwidth and other stats on the F5 during peaks to see if there is some kind of plateau? Also have you checked the logs on the f5 for anything unusual? Have you checked the load on the indexer during peak times? Perhaps it is underpowered for the input load. How about other indexing functionality on the indexer during these times, are you indexing from any other machines and is this not impacted during the issue? (you could search the internal indexes for activity during these times if you have no other inputs) This information might help the community provide assistance. ... View more

Have a look at the transaction command. If your search returns requests and responses with an identifier, all you usually need to do is add something like "| transaction GUIDFIELDNAME" Depending on things such as maximum expected request time and if request and response events have identifiers you could make it more reliable by using additional options like maxspan, startswith, and endswith. This command will add some fields to your results, one of which is duration in seconds, and this sounds like what you are after. See: http://docs.splunk.com/Documentation/Splunk/6.2.1/SearchReference/transaction ... View more

Is this your log data or how splunk is indexing it? If the latter, It looks like your config is not successfully parsing the timestamp entry in these java logs or it is not set to break events on timestamps. If like most of my java logs, the event starts with a line containing a timestamp, you can normally successfully parse this by telling splunk to break events on timestamps and a combination of MAX_TIMESTAMP_LOOKAHEAD and TIME_FORMAT, I also usually specify timezone with TZ=[cont/region]. If you post the first line of an event, we may be able to suggest TIME_FORMAT strings. If your logs are adding timestamps to every line, perhaps you could correct that on the application side. Otherwise you will want to see if the starting line uses a different timestamp format and tune splunk to only recognize that one. ... View more