I'd venture that as long as a forwarder is sending something, it probably hasn't just stopped sending one sourcetype. I'd use the Deployment Monitor app here. If I recall correctly, there is a dashboard that deals with something like "Forwarders that haven't sent any data in 2 hours", or something to that effect. ... View more

Typically it means that your file, in this case, access.log, rolled over. Splunk sees that there's a new file called access.log, but it looks different from the old file that it had been monitoring. It assumes that your log rolled and so it starts reading it from the beginning. ... View more

Yep. You want to look at doing a host override and a sourcetype override. Well, several of each, probably. These links should get you started: http://answers.splunk.com/answers/12439/sourcetype-override http://answers.splunk.com/answers/24769/host-override ... View more

Ok, so based on your answer that you can't change the app to log with a full timestamp, you still have an option, though it's imperfect. In your props.conf file, you can use something like this: [nameofyoursourcetypehere] DATETIME_CONFIG = CURRENT This will cause Splunk to "assign" a timestamp of the current time (i.e. the time the file was indexed) to the events. Very likely, all of the events would have the same, or almost the same timestamp assigned since it would only take a few seconds to index the file. But, frankly, without a timestamp in the log, you don't actually know when things happened anyway. ... View more

Ok. Is your TIME_FORMAT correct? The standard TIME_FORMAT for the syslog sourcetype is: TIME_FORMAT = %b %d %H:%M:%S Which expects a month abbreviation (i.e. Jun, Nov, etc), then the day. Yours expects a timestamp like 12 09 16:27:00, which seems odd. ... View more

If having an actual timestamp is important to you (and I'm guessing it is), you're going to want to have a real timestamp placed into the csv file. What app is logging to that file? Can the logging format be changed to provide a real timestamp instead of just a date? ... View more

Are you trying to define an input with your [source...] statement? Because that's wrong. In a props.conf, the [source...] statement is meant only to define what group of events to apply the props to. You'd want to create an inputs.conf with something like this in it: [monitor:///HOSTS.../*.logfile] sourcetype = syslog And then, if your syslog is properly formatted, you won't even need a props.conf stanza for that. ... View more

index=myindex sourcetype=access_combined source="/path/to/access-brand2.log" EDITED to correct index... ... View more

If both logs are in the access_combined format, you can just tell Splunk that they're access_combined logs. [monitor:///path/to/access-brand1.log] index = myindex sourcetype = access_combined [monitor:///path/to/access-brand2.log] index = myindex sourcetype = access_combined The above stanzas should work. ... View more

shacham, I'm not really sure what you're asking. Maybe you can clarify? If you want to monitor both files, you can do something like this: [monitor:///path/to/access-brand1.log] index = myindex sourcetype = brand1_access [monitor:///path/to/access-brand2.log] index = myindex sourcetype = brand2_access ... View more

If your node logs hard drive failures somewhere (either in a file, or via syslog, or whatever), then yes, Splunk can monitor those vectors to alert when there's a problem. ... View more

Deletion of data (via the 'delete' command) won't increase performance. It's sort of a misnomer. The 'delete' command won't actually delete any data from your indexes, it will only make the data 'invisible' to searches. Cleaning out an index is certainly an option, but a drastic one. If you don't mind losing ALL data from your index, you can go that route. I'd start looking at the underlying causes of WHY your searches are slow. Are you piping everything into one index? Maybe look at separating your data into different indexes. This should make searches (prepended with index= ) run a bit faster. Over what time range are you running your searches? If you're constantly running searches "over all time", then you should get out of that habit. Only run a search over the time range you need. How many scheduled saved searches do you have running? If you're running Splunk on an underpowered server, your ad-hoc search may be contending with scheduled saved searches (or other users running ad-hoc searches) for CPU cycles. ... View more

TIME_PREFIX is used to tell Splunk what comes before the timestamp. Also, in that TIME_PREFIX that I provided, the regex in the first set of parenthesis will match any number of returns or newlines. ... View more

I was going to suggest using followTail, but the docs state that: "* DO NOT leave followTail enabled in an ongoing fashion." I wasn't sure how much of a negative effect it would have to leave it enabled for 8 hours a day. Thoughts? ... View more

No problem. If I've answered your question, please remember to accept one of the responses so the question shows as answered. Thanks! ... View more

With the free license, yes. ... View more

This doesn't seem like a reasonable architecture. What you're describing isn't a forwarder (heavy, or otherwise), but an indexer. And distributed search isn't available with the free license. Your only options here are either to be more choosy about what you forward, so you can stay below the 500MB limit, or pony up for an Enterprise license. ... View more

You have two options here: Turn the first server into a lightweight or heavy forwarder per the instructions in the link I posted before. Turn the second server into a search head, and the first server into a search peer. The second option isn't really an option for you if you're on the free license. I'd go with the first option. When you do, the data will no longer be getting indexed on the first server. The first server will just forward the data to the second server, where it will be indexed, and can be searched upon. For reference, here is some info on the capabilities for the different types of licenses. ... View more

More info about heavy vs. lightweight forwarders here. I understand your question, I'm just trying to determine if you're actually indexing on both servers. If you are, you probably don't need to be. And yes, if you're indexing on both servers, both count against the license. I'm assuming that you've got the free 500MB license on each? If so, then both server's licenses would be getting dinged. ... View more

So, does that mean you converted the first server to a lightweight or heavy forwarder? ... View more

How are you sending the data from the first Splunk server to the second? ... View more

There's nothing built into Splunk to accommodate this as far as I know. You could hack something together that would enable/disable an input via a script, but as soon as you re-enable the input, it would just index all of the data from the time you don't care about. ... View more