About sdwilkerson

sdwilkerson · ‎11-23-2011

Gaurav_a, There are a few ways to do this. The most straight-forward way is to add the "index=foo" line to your stanzas in inputs.conf as described in the docs. For example, if this were for a UF on Windows to collect normal WinEventLogs, and you want to forward events to an index called "windows", you could add the index line to your inputs.conf to make it look like this: [WinEventLog:Security] index=windows You can also achieve the same results via the Splunk CLI commands. There are also more advanced techniques if they suite your use-case more appropriately. Best, Sean

sdwilkerson · ‎11-23-2011

Sushildabare, I agree with Ayn's comment, that a realtime search should show events in realtime. Perhaps there are bad time (or timezone) in your data? Choose "Real Time > All time (real-time)" entry in the TimeRangePicker pulldown" instead of one of the realtime intervals. This should show you data on the screen AS it is received. Examine your timeline. Are the times current? If the data is the volume you expected, but the times/dates are off, then that is the answer to your problem. Of course, the you would have a new problem, which is to fix the time on the source or fix Splunk's interpretation of the time or timezone in props.conf. Best, Sean

sdwilkerson · ‎11-23-2011

Sushildabare, I think you may be confusing inputs.conf with indexes.conf. Inputs.conf handles all inputs (as you mentioned); however, only indexes.conf adjust your indexer's retention policies. So, if Splunk retires data at 3 months there is nothing about that config that would tell it to pull the data in again since that is handled separately. Based on what you said here, all data will be retired when it is 3 months old. If the file in question has not yet changed, Splunk will not know to re-index it automatically. Essentially, your policy told Splunk to purge the file. You can adjust retention policies by index, but not by index-contents. If you want data in this file to be preserved in Splunk for a different period of time than other data, then I suggest you input the contents of this file into a separate index that has its own retention period. Depending on what you get out of this file, you also might want to consider pulling it in as a lookup table (not normal indexed data). Maybe you have a scripted input run once a day, pull this data in to Splunk, then have a search which dumps the data out to your lookup. This would require a lot of re-indexing of this data, but if it is small this is trivial for both the system work and your license. Another idea is to use the splunk-base app getwatchlist if this file is accessible via ftp/http/https and has any kind of standard delimiter. This would allow you to run a regular search in Splunk (e.g. daily) to pull in this list to the search and dump it directly out to a lookup and not have to index the data at all. Best Sean

sdwilkerson · ‎11-22-2011

Hello, Trying to have Splunk monitor standard scan-reports from Foundstone (Vulnerability Assessment Scanner), but repeatedly seeing this in the splunkd.log: 11-22-2011 17:13:26.759 -0500 ERROR ArchiveFile - In archive '/data/splunk/splunk-4.2.4/var/spool/splunk/Monthly-Full-2010-102811.csv.zip': Bad ZIP file This zip file opens fine on the windows system with the built-in zip, and on linux with "unzip." Any ideas what is causing the problem? Is it possible that Foundstone uses a compression algorithm that Splunk doesn't understand and if so, how can we test for this? Any idea on how to get around it besides a scripted input? Thanks, Sean

sdwilkerson · ‎11-21-2011

Masa, Good catch. Thanks.

sdwilkerson · ‎11-21-2011

Good catch on the KnownIssues.

sdwilkerson · ‎11-21-2011

JBSplunk, It looks like Masa contributed a great solution, but I wanted to share what I generally use as well, since it is shorter syntax (less to remember/paste). I use Larry Wall's "rename" perl script. (Remember, Larry Wall is the father of perl). This rename comes stock on debian/ubuntu but it is NOT the same as the CentOS/RHEL rename. To use it on CentOS/RHEL I download a fresh copy from the Internet. Larry Wall's "rename" takes sed-style matches, so the following will work: cd $SPLUNK_DB/targetindex/(db|colddb) (Where targetindex is the name of the index you want to manipulate) START=XXX (where XXX is the bucket number you want to start counting on) Run the following: for i in `ls -rtd db_`; do START=$(($START + 1)); rename -nv "s/\d+$/$START/" $i ;done* The ls with rtd args, sorts reverse by time and only shows the directories, so the oldest bucket will be listed first and therefore will be the first bucket in your list to keep the order somewhat the way it would have been originally. The business with the START thing helps to maintain a counter so you can move through them one at a time. This is important when you are doing both the HOT then COLD DBs, and therefore you need to start on a specific number. Sean

sdwilkerson · ‎11-21-2011

joonradley, I have started seeing this error at several customer sites after upgrading them to Splunk-4.2.4. I never saw it prior to 4.2.4. I believe this is a logic bug of some kind (it looks like a change which is inappropriately measuring for local.meta's existence). The fix is going to be when Splunk puts out a patch. In the meantime, you can create an empty local.meta file in your apps and the error goes away (in the cases where I have seen this). I use a little for loop to implement this solution: cd $SPLUNK_HOME/etc/apps (or $SPLUNK_HOME/etc/deployment-apps) for i in ls -d ; do [ ! -f $i/metadata/local.meta ] && touch $i/metadata/local.meta; done; Reload and all is well. I would love to hear someone from Splunk provide more details on this. Best, Sean

sdwilkerson · ‎11-21-2011

Vladimirc, The "DeploymentServer" and "DeploymentClient" is used to control configurations on remote systems, but isn't the utility/conduit to how logs are sent. Did you intend to setup the DeploymentServer as well?

sdwilkerson · ‎11-04-2011

Cloudharmony, There are a few things to consider that might perk up your result times: Start your search using indexed fields (e.g. sourcetype, source, host, and/or index) to prevent Splunk from having to waste time looking at irrelevant data If this is a query you will perform often, create a summary search to run at some set interval (e.g. 10 min) then report across the summary data. Beyond the above, yes you do take performance hits with various splunk analytic commands and there is some guidance to help improve this in the Splunk docs. Sean

sdwilkerson · ‎09-28-2011

Curtgan, Yes, this isn't the right place, you should really have started a new question. But the answer to your question is, yes, transaction doesn't care so long as the time settings and field are right.

sdwilkerson · ‎09-18-2011

fk319, I agree with jlunk that the problem might be your disabled line. Since you do not control the remote systems, it is possible that a configuration is not exactly as you would like it to be. If you choose to force/override the metadata settings you can do this. If the remote systems are Universal Forwarders, then you can install your props/transforms on the receiver of their data to override the index metadata. Is this where you have setup your props/transforms referenced above or did you put it on the Universal Forwarder doing the collection? Note, for troubleshooting, if you are not doing encryption, you can do a packet-capture on port 9997 on your receiver to look at the data coming from these systems. You will be able to clearly see the metadata settings in the pcap and can see if they are being set by the sending host. Sean

sdwilkerson · ‎09-14-2011

Mattlemay, You configured Splunk to monitor (read: Index) all of /var/log and now you are disappointed that it did what you asked it to? Generally you want to explicitly state what you want Splunk to do. For instance, if you want to monitor all of /var/log/apache then configure it to do that and it will.

sdwilkerson · ‎09-14-2011

Branden, It is hard to say exactly, since you didn't include the rest of the stanza and we don't know what your monitor line looks like, but generally Splunk pulls the entire PATH of the files it monitors (this typically shows up as the "source" field. Therefore, your regex, "^esx-*" assumes that the beginning of the line starts with "esx". If this is windows, the beginning of the line is probably something like C:... and if it is Linux it might be something like "/var/log". It is unlikely the path to the monitored file will begin with "esx" without something in front of it. Try removing the ^ since that might match, or stating the explicit path that should appear in appropriate regex format. Best, Sean

sdwilkerson · ‎09-14-2011

Triptrops, In your step above: "./splunk add forward-server :9997", what is the name/address of the receiving host (i.e. server1) that server2 should use? Note: That IP should go before the :9997. When you run $SPLUNK_HOME/bin/splunk help add, you will see this example in the output: ./splunk add forward-server bologna:9997 In this case, the system bologna is the receiving host. Set this correctly, and it will probably work. Also, as a side note, if you are going to use the SplunkLightForwarder, you will probably be better off using the Splunk Universal Forwarder (a different installation package). Sean

sdwilkerson · ‎08-23-2011

What fields do exists in the third index that might be used to unite those events with events from one of the first two indexes? A subsearch or double-transaction might work.

sdwilkerson · ‎08-05-2011

Is there a recommended best-practice for collecting the version of windows that a certain Splunk-instance is running on? In this case, the Deployment Server will be used to push out this "version app" to the enterprise so we can centrally-report on stats by Splunk version. Thanks, Sean

sdwilkerson · ‎07-27-2011

LCM, The Deployment Monitor runs many searches and then writes summaries of these searches out to a handful of special summary indexes. Using it in a Splunk Distributed environment (as you are doing) is very useful but you must consider a few things. The Deployment Monitor should be running on just the search head. If this Search Head has a default configuration, it will be storing the summary data locally (on indexes that are on the search head and were created by the indexes.conf that ships with Deployment Monitor). There is nothing wrong with doing it this way, but there are other ways (mentioned in a minute). If your system is setup this default way, then you should check the summary indexes on your search head to see if they contain fresh data. If they do contain fresh data, that means it is being collected, but you just aren't seeing it and that is a different issue. If they do not contain fresh data then your app is not collecting this data. If it is not collecting the data then look into why this might be: Check the scheduled searches specific to the Deployment Monitor and see if any are scheduled to run, If any searches ran recently click on the "view recent" and see if there are results there, Look at one of the queries made by the deployment monitor that it uses to build the summaries and see if you get any results when this runs. An alternate configuration that I prefer to use for a normal distributed Splunk environment is to have your search head forward its events on to your Indexers (or preferably a cluster of Heavy Forwarders for large-scale deployments). In this environment, your Search Head would run the query against the indexers, then it would generate the data that should be indexed in the summaries and forward this data on to the system configured in your outputs.conf. This configuration would allow the data collected to be sent across all of your indexers and no data would be sitting on your search head. If you do this for the Deployment Monitor app, you must remember to create these special summary indexes on your indexer otherwise when the data gets to the indexer, it will not be prepared for it. Here are very granular details. Best, Sean

sdwilkerson · ‎07-25-2011

Dimitry, I see a problem in your config and there could be several things preventing access. First, the problem. I noticed in your included inputs.conf that you force events discovered in "/var/log/messages" to go to index=_internal. The _internal index is very special and generally reserved for Splunk-internal logs (hence the name). If you omitted this line entirely, a default instance of Splunk would automatically place new events into the default index which is index=main. You might want to omit this line or create a new index and use that since _internal is definitely not the right place for your data. On to your reported issue here are some bullets to consider or help you troubleshoot. Is the "server" configured to listen on port 9997? If not configured to listen, then run this command on the server/indexer: $SPLUNK_HOME/bin/splunk enable listen If you did configure it to listen, then attempt manually to connect from a sender (forwarder) to the receiver (indexer) by doing something like this on the sender: telnet 172.16.40.116 9997 This telnet is to see if you have TCP access and connectivity from the sender to the receiver on port tcp/9997. If you see your telnet session "connect" and go to a blank screen then this test was successful and you are having some other problem. However, if you see your telnet session hang at something like "cannot connect to host" or "connection to host refused" or something like that, then you might have an issue with a firewall, router, or access control either on or between the sender and the receiver. On the receiver (indexer), does the $SPLUNK_HOME/var/log/splunk/splunkd.log show the connection attempts from the sender(s)? Do you have encryption or compression configured on the side of the receiver but not on the sender? Note: if you did, this would be configured on the "input.conf" on the receiver. Sean

sdwilkerson · ‎07-21-2011

With Splunk-4.2 (license manager), should a purchased HA-license go into the normal license pool on the production license manager along with the real license? Is this a tolerated use of the license? Doing this presumes that the Indexer(s) in the DR (Disaster Recovery) location would use the production license manager for its licensing needs. Of course this raises a different concern, which is, if you are deploying a DR Splunk environment is it wise to have your license-manager be in the production facility only? This seems to me to answer the first question, and perhaps each site should be stand-alone to ensure the DR facility can function without the production facility. Thoughts? Sean

sdwilkerson · ‎07-19-2011

Bhiley, Yes, there are limits in splunk, which you may be hitting up against. See, in your search you are telling Splunk to RETURN all of this data and not just count it. Splunk has many analytic features, such as "stats" that have arguments like count. You probably want to do something like this: search = index=tal | stats count If you want to be sure to just run this for the previous day you can use the timepicker in the search or do this: search = index=tal earliest=-24h@h latest=@d-1s | stats count If you had multiple indexes you can do ... | stats count by index Find more about stats here: http://www.splunk.com/base/Documentation/latest/SearchReference/stats Best, Sean

sdwilkerson · ‎07-19-2011

Do the indexes "testindex1" and "testindex2" exists? Did you create them either through the UI, cli, or by editing indexes.conf?

sdwilkerson · ‎06-23-2011

gfriedmann Any app dir should work (taking into account precedence). Just to be clear, you set the value in "seconds" and not "minutes," correct?

sdwilkerson · ‎06-23-2011

Thanks gkanapathy. Using clientName means blacklists/whitelists will still need to be used, but only on the base install package stanza. This will make the serverclass.conf cleaner, but still has the same complication of managing the *lists. Correct? Also, this still requires separate deployment-apps for each "index" variation. Is there a simple way around this, that I am missing?

sdwilkerson · ‎06-22-2011

An enterprise network has many sub-networks each with UniversalForwarders forwarding to a central pool of Indexers. The data from each sub-network should be contained within it's own index, e.g.: index=site1-windows index=site1-unix index=site2-windows index=site2-unix Since managing many hundreds of IPs in the blacklist/whitelists on a single Deployment Server is painful, what is the recommended approach to ensure the deployment-clients from one subnet properly assign the data to the correct index? Here are two possibilities, but am certainly open to other suggestions (or a confirmation that one of these is best-practice): A separate Deployment Server for each sub-network. This is a lot of extra work and duplication, but allows use of "machineTypes" and requires less ongoing *list maintenance. Parsing configs on the Heavy Forwarders to assign the correct index based on subnet via regexes Thanks in advance.

Posts	125
Solutions	18
Karma Given	82
Karma Received	108
Member Since	‎04-14-2010

Online Status	Offline
Date Last Visited	‎03-11-2024 09:39 AM

The Splunk for Symantec app (v 1.0.3) doesn't disp...

The Splunk For Bluecoat view "bcoat_overview" does...

Intentional stopped service in MSExchange

Stash Files contain binary data

Grouped Search Results by IP

Frequency lea updates pointer file

Plot time lag trend with scheduled search

Splunk fails to monitor zip file

Collect Windows Version

HA License Usage

Re: Splunk Forwarder

Re: Realtime search is very slow

Re: Retirement and archiving period for one file

Splunk fails to monitor zip file

Re: How can I find all duplicate bucket id's that ...

Re: Failed to install app

Re: How can I find all duplicate bucket id's that ...

Re: Failed to install app

Re: Setting up Windows Splunk to accept universal ...

Re: Performance Expectations

Re: How to build a transaction from multiple, some...

Re: Splunk Receiving

Re: Really quick and basic example?

Re: Can't get a blacklist to work - please help!

Re: Cannot see host on the Splunk server

Re: How to build a transaction from multiple, some...

Collect Windows Version

Re: Splunk Deployment Monitor 4.2.2 is not working...

Re: Forwarder: TcpOutputProc ... Connection closed...

HA License Usage

Re: Reporting the number of events in an index

Re: Distribute data from one source to different i...

Re: Deployment server strategies

Re: Index assignment for related sub-networks

Index assignment for related sub-networks

Are you a member of the Splunk Community?