It's only returning the count over the past 6 days, how should I test this? Do you recommend I change this ..| where count > 0.5 so an alert triggers at 50% of the current average? Would it be possible to create a barchart showing the count for each day (over the past 6 days) and have a trendline showing the 6 day average. Then I can have the alert go off if it goes 50% higher? ... View more

To add onto this with an example From [Aug 5th - Aug 10th] the average number of errors for the day was 90,000 Say on August 11th the average is 140,000, we are now 50% more than what our average was from [Aug 5th - Aug 10th] and an alert is sent ... View more

Yes very close. I want an alert set on a cron schedule to run every 6 hours (It was originally 1 day but now I want it to be 6 hours). So the alert will have an average from [Aug 5th - Aug 10th] and if at anytime that number goes 50% above the ~6 day average then an alert is sent out ... View more

If this was helpful then please accept my answer ... View more

So you have from=offline state start, to=offline state end, in the same event.. If you're going to group them with a transaction command then they need to be separate events. Below I pasted your example but removed the "offline state end" from the first event and removed "offline state start" from your second event. So in this case you can use a transaction command to group those independent events into one event. [2015-08-11 00:38:53,747] INFO tracking.WidgetStateMachine [tid-169]: cmd=TRANSIT_STATE, device_id=112233445566, from=offline state start, action=timeout [2015-08-10 23:33:47,244] INFO tracking.WidgetStateMachine [tid-339] [request_id=1122334455666:6:0] : cmd=TRANSIT_STATE, device_id=112233445566, to=offline state end, action=connect index=whatever | transaction startswith="offline state start" endswith="offline state end" | stats sum(duration) as total_offline | eval online=100*(86400-total_offline )/86400 Since you have both "offline state start" and "offline state end" in the same event already then you do not need to group them together. Is the device_id unique to each start/end? You need a unique field to tie the first event to the last event ... View more

Did all the temp files get deleted when you uninstalled splunk? ... View more

Good explanation.. I'm doing the extractions now and we expect a large increase in events in the future..Even though it's not affecting performance much right now, I don't want to hurt myself in the future ... View more

Yes, put in your search, then go to 'Save As'>Alert Give it a title, description, Alert type should be 'Scheduled', time range should be 'Run on a CRON Schedule' Earliest = -15m@m Latest = @m Cron Expression = * * * * * Trigger if number of results 'is greater than 45' ... View more

Yes, it's defiantly possible.. Instead of doing a stats, you can just do a table command if you want to add more columns source=myapp test123 | table type brand color Or if you wanted it to calculate the type and add a table, you could do.. The eval command will create a new field and do the math (size/occurance). The result of the division will be the new field "type" source=myapp test123 | eval type=(size/occurance) | table type brand color ... View more

Do your events have a unique identifier tied to the "offline state start" and "offline state end" to represent that they are pairs? If not then I suggest combining those logs together as 1 event at index time rather than search time. To do this you will need to go on the indexer and go to Splunk_Home/etc/system/local and edit your props.conf file to include those independent events as 1. Share some sample data and I'll help you out ... View more

Thanks for sharing that link! This solves a huge problem we have with dashboard performance!! ... View more

It looks like you have bad line-breaking.. If this is the case, then you should fix it at the index level compared to using a transaction command at the search level to maximize performance. Is the current line-breaking correct? ... View more

The transaction command only groups independent events together.. So if you have 2 events, 1 is a request and the other is a response. Event 1 has the words "request" and event 2 has "response", you can then jon those 2 events into 1 event by doing this index=whatever | transaction starswith="request" endswith="response" Once you have 1 event, you can then easily find the duration between then 2 events.. I don't think this applies to what the original question stated as it wasn't clearly defined. It looks like you want to take 2 searches and combine them together, then do an | eval and subtract those fields and plot the results vs time, is this correct? If so, then give me some sample data and I'll fix your search ... View more

If you do not specify the index name in your inputs.conf then it will automatically be in index=main ... View more

Then it is already picking up those logs.. If you did not define the index on the forwarder then it's in index=main which is the default index. ... View more

When you say "search server" do you mean the indexer? You need to define your index name on the forwarder and not the indexer.. So say you have a server called 'SRV-Aux01' you want to collect log files from.. You will then go to that server and install a forwarder on it, you will then go to etc/system/local/inputs.conf and define your stanza which includes your index name. If you wanted to define the linebreaking or anything related, you will do that on the indexer ... View more

What exactly are you trying to extract? Do you want whatever is where the # sign is in your example? You could use a \s+? to capture either a single or double space.. Give me a line or two of exactly what you want extracted ... View more

Do not modify your .conf files yet.. Go to the server which you want the data off of. Then install a forwarder (See below). Once you have that forwarder installed you will then go to your inputs.conf (etc/system/local) and you will create a stanza.. Your stanza should look like this (This should be defined on the forwarder, not the idnexer) [host::hostname] sourcetype=log4j index= YOUR_Index_NAME http://docs.splunk.com/Documentation/Splunk/6.2.4/Forwarding/DeployaWindowsdfmanually ... View more

Try this and let me know if it works.. It may need some tweeking as its untested "New apps retreived" OR "New apps generated" | stats count values(Apps_retrieved) values(Apps_generated) | eval Diff = Apps_retrieved - Apps_generated | timechart count(Diff) span=1h ... View more

If I understand this correctly, you want to find the difference between timestamps which will show you how long an event took to process? You first need to see what the events have in common, usually they have a unique identifier tied to each request/response pair. Then you can pipe it into a transaction or stats command which will then group them. Then you will pipe it into a timechart If it doesn't have a unique identifier and is in the same index, you can then use startswith="start" and endswith="ends" index=whatever | transaction startswith="start" endswith="end" | timechart avg(duration) ... View more

There's a few ways of doing this.. You can either upload them manually into Splunk (See documentation below) or you can set up a forwarder to automatically upload data into Splunk as it comes in.. I'd recommend using a forwarder http://docs.splunk.com/Documentation/Splunk/6.2.4/Data/Uploaddata ... View more