I meant them both separately. You could use the supplied script to gather the data but I meant just looking at the system load directly. On SoS there is a dashboard called "Indexing Performance". This can help to give an indication of if the system is under load, by default the middle chart will display four of the main internal queues Splunk uses, the lower one called indexQueue is the point where Splunk writes the disk. If these are starting to block then something is impacting Splunk somewhere. The timeout is hardcoded and cannot be altered. ... View more

Did you run the upgrade as root and then try to run Splunk as the Splunk user? ... View more

Oh wow, this is a blast from the past! I can't recall how I solved this but I just use sideview utils now 😛 ... View more

Have you tried setting the index= on the indexer side? ... View more

We could really do with some example data to really help but whatever you do you aren't going to be able to compare them without performing an extraction on the host name in the event. Putting this in transforms/props isn't at all tricky but will leave you with a host field that is extracted each time you run a search (without doing rex). So, there is something in the event that links it to a host? To match these you need to have a common field. Forget inputlookup for now, I don't think thats the right command for what you want to do (it just throws the CSV onto the end of your results). Lets say you had an event like this in mydata.log; 192.168.2.1 01/01/12 14:30 Error: I'm an event! With no meaning! Danger! Now you want to match the IP to the host in your CSV. Assuming you have the IP being extracted somewhere by rex or a rex in transforms/props, lets build a CSV... ip,hostname 192.168.2.1,DC and we'll call this hostnames.csv Then in the search I would do a search for; source=mydata.log | lookup hostnames.csv ip OUTPUT hostname This would look for hostnames.csv either in the lookup directory of the current app you're in or for a global csv of the same name, it would pass in the ip and for any matches it would throw out the hostname. This can be taken a step further by writing this straight into transforms/props so every time you run a search it does this, thats where the power of Splunk comes into play. You could also do this via DNS if you really wanted but it depends on the use-case. http://blogs.splunk.com/2009/12/15/reverse-dns-lookups-for-host-entries/ Post more details if this is not what you were talking about. ... View more

If you are replacing any files within Splunk just be wary that you will need to replace these any time you upgrade Splunk - it will overwrite them each time otherwise. ... View more

Even without a support contract you can still submit a support request via https://www.splunk.com/index.php/submit_issue The only difference is that you have no guaranteed SLA but someone at Splunk will eventually read it. Best bet is to include the bug detail in the subject to grab their attention ... View more

This is a known issue, bug SPL-58292 MainTailingThread crashes splunkd with a message that says 'Assertion failed: bytesToHash < 1048576' (SPL-58292) Whenever I hit a new problem I always hit the known issues first to be safe; http://docs.splunk.com/Documentation/Splunk/5.0.2/ReleaseNotes/Knownissues In the meantime, you should contact Splunk support for more help ... View more

Have you checked the system load? Looking at IO, CPU and MEM (although mainly CPU). Also in SOS are any of the queues filling up? ... View more

Where to even begin... Firstly I think this shouts to me that you haven't read any of the documentation and that you're trying to hack Splunk into doing something you want by diving into the code... STOP! Go back, read the tutorials and learn! Firstly, what you have pasted above is some advanced xml. Have you tried the simple xml yet? In the above paste its a HiddenSearch module which has a required search param, inside here you paste the search you want to run. You ask how to search a csv? Well.. is it a lookup or indexed? if its indexed then you search it like you would any other indexed data, if its a lookup then you can use the |inputlookup command to pull it in to then search. The more detail you give the better the answer, also the more you try to learn yourself the easier you will find this 🙂 Some links; http://docs.splunk.com/Documentation/Splunk/latest/Tutorial/WelcometotheSplunktutorial http://docs.splunk.com/Documentation/Splunk/latest/Tutorial/Aboutgettingdatain http://docs.splunk.com/Documentation/Splunk/5.0.2/Viz/OverviewofSimplifiedXML http://docs.splunk.com/Documentation/Splunk/5.0.2/SearchReference/Inputlookup ... View more

What makes you believe that the indexes are corrupted? The timeout between Splunkweb and Splunkd is hardcoded and as I recall was around 30 seconds. This means that if the system is under load it can cause splunkd to not reply in time. Quite possibly it could still be doing a bucket conversion or a spike in indexing. Download the Splunk on Splunk app and try to install that (if you can). Plus checking the logs as MuS said is a good start ... View more

Could you paste your XML/Search so we could have a test? 🙂 ... View more

Although Splunk updates the version of Python that ships with Splunk, as I understand it is still a custom build for Splunk and as such some libraries may not be updated or supported (I had some issues a year or so ago with a missing library). For MS SQL connectivity, depending on what you want to do, you may find this app helpful; http://splunk-base.splunk.com/apps/50803/splunk-db-connect ... View more

By default Splunk will build a CRC from the first 256 bytes of a file, regardless of if it is a CSV or other. Is it possible that even though the contents aren't changing some other heading information is changing? If you have a look at - http://docs.splunk.com/Documentation/Splunk/latest/admin/inputsconf You can edit the amount of bytes it builds the CRC from by changing initCrcLength. Otherwise you may be better to actually store the CSV in a lookups folder and search it as a lookup? This way you could build a dashboard that uses | inputlookup to pull in the CSV and then do a search against that for certain criteria. ... View more

Some basic tips which you'd need to clarify if you've tried first; Changing the inputs.conf requires a restart for the changes to take effect When you say the index is successfully showing data from Linux hosts, is this through a dashboard or by you searching index=otherindex in a search window? By default you won't probably be searching this index when you run searches unless its been added to your role. Once a forwarder has sent data it won't re-send it unless you clear its fishbucket out, but you should still receive new events - if there have been any ... View more

Have you tried clicking on the Download button to the right? You then have to install the app through Splunk by going to Manager -> Apps then click the button that says install app from file. Navigate to the file you've downloaded and like magic its there... However, if you're struggling with even this step you may want to stop and read the tutorials before trying to do any UI dev ... View more

This is how I found it; 1) Selected the search box in the top right 2) Selected the option to search apps 3) Pasted in, UI Examples for 4.1+ 4) Clicked on the link for UI Examples for 4.1+ 5) Arrived at http://splunk-base.splunk.com/apps/22333/splunk-ui-examples-app-for-41 Simples ... View more

But to be clear, you do this through the same login page that you would see at the moment. ... View more

That was my point... the best part of it relies on the remote data collection, without this the app can only work with the event log data you may be sending back to it 🙂 ... View more

The issue is that without finer detail in the timestamp there isn't really any way to ensure it appears as it does in the log. Can you increase the timestamping? ... View more

Perhaps try changing the sort _time to... | eval indextime=_indextime | sort indextime This "may" work better than _time, if not you're just going to have to increase the timestamping on your logs as there is no guarantee it arrives in Splunk in the same order, thus Splunk doesn't know the original order in the log. DISCLAIMER: This obviously won't work if you have historical events or any other funny time related business going on. ... View more

Have you had a look at the download page? There is a .deb version available for Splunk which will install on Ubuntu and there is a fully detailed installation manual on docs.splunk.com which I recommend you read as it gives you things to prepare and how to complete an install. The bit related to .deb is here; http://docs.splunk.com/Documentation/Splunk/4.3.2/Installation/InstallonLinux#Install_from_a_Debian_DEB_package (But you should really start from the front page of the manual) ... View more

and how does it appear in the log file? ... View more

Yes, or perhaps to tell us how it appears in the log file so we actually understand how you want it to look 😛 Its pretty hard to replicate it without knowing! Although Lguinn is probably bang on the money with a sort by time ... View more

A quick search of the docs page reveals that it was fixed in 4.3.2. http://docs.splunk.com/Documentation/Splunk/4.3.2/ReleaseNotes/4.3.2 If you ever want to know about known issues or fixes per version always have a look at the release notes first. ... View more