About Drainy

Drainy · ‎11-03-2014

The other answer misses the potential issue of a forwarder re-forwarding data that has already been seen. In these sorts of setups you need to take a step back and consider the following; 1) If these logs are critical, consider how they are distributed between systems. A common setup may be to use a load balancer to split syslog between two servers, these can generally ping a port to check that the service is available and so if the forwarder goes offline it will send all data to a secondary server 2) Again, if they really are critical you could always setup two forwarders sending to two different indexes on the same indexer. This way you could maintain a primary and secondary copy 3) No service I have ever seen runs at 100%, or even has 100% as a reasonable SLA. Downtime should be expected at some point, normal proceedures to handle process monitoring and a forwarder going offline should already be in place. Some monitoring tools can also be configured to automatically restart a failed process. In this case I would simply accept that as a best endeavor, short of custom scripting some hideous to maintain piece of custom sticky plaster work 🙂 Bear in mind that the forwarder will just pick up where it left off once its restarted.

Drainy · ‎11-03-2014

Splunk already has a field for when data is added (or indexed to be more accurate) _indextime. You could flip this around and set _time at index time to be the value of Date Added, have a look at props.conf to set this up; http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/Propsconf Its pretty straightforward and your date format should be understood without issue. This way you can then search by _indextime and _time, although if you want the performance boost of filtering on _time with the expectation that this is the date added, then this option isn't for you.,Just to throw another option out there, Splunk already has a time field for when data is added - _indextime. So why not just override _time to be Date Added at index time? Have a look through http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/Propsconf for all the options but its pretty simple to configure and you may find it a better solution in the long run. If you want the performance boost of filtering on _time being the time added then this won't be for you but if you're going to search for both values, it should help.

Drainy · ‎03-21-2014

This is a permission error. OSError is returned when Python can't get permissions to access, change or create a filesystem object. Double check the permissions to make they are as you expect. Bear in mind that this is invoked by the python interpreter that comes with Splunk and can sometimes throw some wobblers with permissions errors like this.

Drainy · ‎01-06-2014

ok so time configuration in props is about how it pulls the timestamp from an event. Its not related to how Splunk handles rolled logs, thats done by the CRC of the first 256 bytes of a file (or something like that. Try crcSalt = in your inputs definition for that file, it will force it to use the filename as the crcsalt which will always be unique as long as its a unique filename

Drainy · ‎01-06-2014

Did Splunk fire a violation/warning for going over the limit? Sure, you can raise a ticket but it will be treated as a "Community" ticket - no guaranteed response time and it will always be at the bottom of the support pile. But you can still raise one 🙂

Drainy · ‎01-06-2014

hmm, I'm not sure if you're confusing problems here? Setting the timestamp stuff may be useful if you're having issues with timestamps. But is that the problem? I thought the issue is indexing/reindexing files? You don't need to configure a CRC value as there is a default, if you want it to read every file you can set the crcsalt to to use the filename as the CRC but you would need to clear your data everywhere otherwise you'll end up with duplicates

Drainy · ‎01-06-2014

So... its not really a simple matter of "converting" a SQL query into Splunk. If you could provide some sample data and what you're trying to achieve then someone may be able to help. It doesn't look too complicated though from what you've pasted above, what have you tried so far?

Drainy · ‎01-06-2014

It sounds a bit like you've reverted to the free license and its trying to log in. Are you trying to hit the login page through the URL? Perhaps just try the URL:PORT of the server? What license do you believe you are running?

Drainy · ‎01-06-2014

I would suspect that this could be one of a couple of things (that I can think of). How tight is your control of searches on the box? Is it possible that users have created lots of subsearches or real time searches and used the map command? I can't recall of the top of my head how this would appear in the job manager but it might fit the pattern. Another option is that you have a couple of dashboards which have some oddly created searches that are impacting in v6 but weren't in v5? Can you do a search through your audit/internal logs to see where these searches are firing from.

Drainy · ‎01-06-2014

Ah, once you've consumed a file Splunk won't re-read it, even if you add a new output. On the forwarder you'll need to run a command, splunk clean eventdata. This will delete all its internal "trackers" of what it has and has't read. This will cause it to re-send everything. So if you do this remember to run the same command on your indexers so you don't end up with duplicate data... its a bit of a clean everything and start again approach. Also bear in mind that on a UF that command will reset the admin password to default, thats not the case with an indexer

Drainy · ‎01-06-2014

Also, it might be worth being sure to change the default password and use IPTABLES to prevent access to it (if you're running on a brand of *nix). I had similar concerns a while back

Drainy · ‎01-06-2014

So I have a few thoughts here, firstly have you had a look at the size of your index on disk to see if it reflects any issues? Secondly, what version are you running? Finally, if you're pretty certain that you've checked everything and it really can't possibly be any crazy log data or wild whitespacing in a custom log - raise a support ticket. The licensing is fairly baked in so short of you actually consuming 250GB+, you'll be best putting your efforts into creating a diag and getting a support case rolling.

Drainy · ‎01-06-2014

Link provided 🙂 In addition to the above, I'd ask here first. Unless its a production impacting issue you will find people here quick to answer, helpful and above all else it gets you involved with the community.

Drainy · ‎01-06-2014

Bonjour! Have you already indexed these logs previously and not realised? Generally logs will be filled with all sorts of wonderful data throughout the day which Splunk gobbles up, then at midnight a script rolls the logs and starts afresh. Splunk recognises this by taking a CRC of the start of each file, this way it stops it re-indexing old data and producing duplicates; http://docs.splunk.com/Documentation/Splunk/6.0.1/Data/Howlogfilerotationishandled

Drainy · ‎10-01-2013

No worries, I've converted the comment to an answer so feel free to click the tick and accept it if you're feeling generous 🙂 Hope you find the problem! (Also, switch to TCP!)

Drainy · ‎10-01-2013

Those results look alright though? thats just the latency and its pretty small - unless theres other bits you haven't pasted 🙂 I have some UDP data and sometimes if theres a blip in the network then latency can be introduced, if Splunk searches before the data arrives then it will rightly think there is an issue.

Drainy · ‎10-01-2013

Looking at; http://docs.splunk.com/Documentation/Splunk/6.0/Forwarding/Compatibilitybetweenforwardersandindexers You will just miss out on two new features, but will work without a problem

Drainy · ‎10-01-2013

Are you sure that the data arrived when it should have? My next thought would be to run something like sourcetype=blah | eval IndexTime=_indextime | eval diff=_time-IndexTime | timechart avg(diff) or something similar for your data over the hour that you experienced problems to see if the events did arrive, but perhaps were indexed later due to latency or some other hiccup

Drainy · ‎10-01-2013

Just to be clear, are you saying that the original source data kept arriving during this time? Is it not the case that you were missing events? Since you're relying on UDP its quite possible that the application could be available but no log data was received

Drainy · ‎10-01-2013

Having a quick skim I think you're missing; _TCP_ROUTING = splunkindexer_9997 From your inputs statements; E.g. [WinEventLog:Security] disabled = 0 start_from = oldest current_only = 1 checkpointInterval = 5 _TCP_ROUTING = splunkindexer_9997 #index = test_index

Drainy · ‎10-01-2013

Are you saying that its now asking for a Splunk user/pass after passing your LDAP authentication?

Drainy · ‎10-01-2013

I'm not entirely sure what you're asking. But to try and cover some, there is a v6 outnow, I assume it has some updates to the REST API and the docs have been updated to cover v6; http://docs.splunk.com/Documentation/Splunk/latest/RESTAPI/RESTcontents

Drainy · ‎09-30-2013

your website 🙂 So if you want to create client-side javascript to connect to your node.js server side stuff to run searches and view visualisations etc. The client-side stuff must be somewhere for users to load and view within a browser

Drainy · ‎09-30-2013

Did you click on the Documentation tab? it has details on how to install and configure the app. To get your data you could configure syslog to output to a listening port on Splunk and just define a tcp input, but yeah the better and more secure/reliable way would be to just install a forwarder and let it handle everything 🙂 There is a contact link on the app so if you do get stuck it might be worth firing a message off.

Drainy · ‎09-29-2013

I've been trying to get a proper link for a manual I found but my laptop really doesn't want to play nice, basically I tried cigesm syslog in Google and the first link (for me anyway) was a PDF manual that seems to indicate that it does support syslog. Is this the same device? A quick search around seems to indicate that similar versions support logging to syslog so it would be odd for it not to. If it doesn't then I believe there is another piece of software that "some" cisco devices use to gleam some data from them but I'm not particularly familiar with it. Generally if you can't push from the device natively then you'd try and get a forwarder onto there to handle the push for you. It might be worth raising a call with Cisco to discuss if you've got an active support contract.

Posts	975
Solutions	127
Karma Given	499
Karma Received	987
Member Since	‎06-01-2011

Online Status	Offline
Date Last Visited	‎06-05-2020 02:03 AM

Appending Sparkline through a JOIN

New PDF Generation - Limitations?

Splunk v5 Clustering and HA

Updated Splunk Visio Stencil or Deployment Icons

Updating Splunk-Base Email

Writing Out to beforeLabel on SingleValue

Shared XML Between Dashboards

Multiple Canvas Flash Timelines within a view

Splunk Install Error on Windows

App Development - Setup.xml not found

Re: Splunk Forwarder fail over configurations

Re: How to convert a timestamp field to epoch form...

Re: Splunk WebFramework error

Re: How to Index Rolling Log Files

Re: Splunk went loco... reporting it indexed 250+ ...

Re: How to Index Rolling Log Files

Re: SQL to Splunk Query conversion

Re: LicenseRestriction: [HTTP 402] Current license...

Re: Mysterious realtime search generating 100s of ...

Re: How to Index Rolling Log Files

Re: Disable the UF web service

Re: Splunk went loco... reporting it indexed 250+ ...

Re: splunk support for customers with license

Re: How to Index Rolling Log Files

Re: Search time range suddenly causes search to re...

Re: Search time range suddenly causes search to re...

Re: Upgrade from 5.0 to 6.0

Re: Search time range suddenly causes search to re...

Re: Search time range suddenly causes search to re...

Re: Heavy Forwarder Configuration

Re: LDAP Authentication Splunk Free broken since u...

Re: REST API Modular Input - Splunk6 - problems?

Re: Help with javascript SDK

Re: getting security onion data into splunk

Re: CIGESM setup