In splunk 4.3.1, the sendemail.py script updates the filename with the timestamp. see line 528 filename = '%s-%s.pdf' % (pdfviewfn[:50], datestamp) ... View more

In that case change the search. by example calculate the number of events per second per source (based on first and last timestamp, over a 10 min period maximum, with 2 min delay) earliest=-12m latest=-10m | stats count first(_time) AS oldestTime last(_time) AS recentTime by source | eval ratio=count/(recentTime-oldestTime) | where ratio>50 | table source count ratio alert on the number of results (the number of source with ratio > whatever) ... View more

Michael, you are the pioneer who discovered the issue with the license manager. ... View more

No, raw format is supposed to contain only the raw event. (the content of the hidden field _raw) If you want more details, try to build a table, with the fields you want. mysearch | eval raw=_raw | table _time host source sourcetype raw Or edit the raw event to add the fields mysearch | eval _raw="time="._time." host=".host." sourcetype=".sourcetype." source=".source." raw="._raw ... View more

comment about scalability of the deployment-server http://splunk-base.splunk.com/answers/12384/hardware-requirements-for-deployment-server-in-large-deployment ... View more

Hi, you are correct on every point : configure the indexer first install on 😧 if you prefer. (if needed you can install on C: and change the path to the indexes to be on D:) (each default index has up to 500GB size) configure a port to accept data from forwarders (default is 9997) install the universal forwarder on the servers to monitor specify the address/port of the indexer during the install in the wizard, or after in the inputs.conf config file, define a monitor input on the folders. [monitor://C:\path\to\my\fodler] If needed, add a white/blacklists to narrow the files to monitor. If you want windows event logs, use the WMI inputs. For the size requirements, it depends of your retention policy, and of the volume indexed per day, if you want you can run a test for a week and see. ... View more

Very hard to guess. My advice, configure a standard one, to forward syslog to a splunk instance in a single index, let it run a week and do stats of volume per day. Then extrapolate for 1000 time more devices. ... View more

FYI, if you also want to calculate duration of unclosed transactions, this is possible with an eval. mysearch | transaction Hostname, LBG_User startswith="checkout" endswith="reset" keepevicted=true | eval duration=if(isnull(duration),now()-_time,duration) | table _time duration _raw Beware, the function now() may not be compatible with real time. ... View more

Sorry I missed this part. So the issue may be regex failing, did you tested it on splunk search, on the sourcetype with the rex command ? Here is another possibility : use sed in props instead of regex in transforms see sedcmd http://docs.splunk.com/Documentation/Splunk/4.3.1/admin/Propsconf in props.conf [WinEventLog:Security] SEDCMD-cleanwindows = s/[rn]+(This event |Note: ).+//g ... View more

This is a known issue on windows with 4.3.0 and 4.3.1 : Fix to come next version. see http://docs.splunk.com/Documentation/Splunk/4.3.1/ReleaseNotes/Knownissues#Windows-specific_issues On Windows, upload a new app from the manager returns "error processing the upload" , changing a server to license-slave fails for "invalid string uri" and the sendemail script and email alerts fail. This occurs only if the user is not using the default server time zone in his or her profile. Workaround: change the user profile time zone to use the default server time zone, or do not specify a user time zone. (SPL-48993) ... View more

This is a known issue on windows with 4.3.0 and 4.3.1 : Fix to come next version. see http://docs.splunk.com/Documentation/Splunk/4.3.1/ReleaseNotes/Knownissues#Windows-specific_issues On Windows 2008, upload a new app from the manager return "error processing the upload" , changing a server to license-slave fails for "invalid master uri" and the sendemail script and email alerts fail. This occurs only if the user is not using the default server time zone in his or her profile. Workaround: change the user profile time zone to use the default server time zone, or do not specify a user time zone. (SPL-48993) ... View more

You are encountering a know bug for splunk 4.3.0 and 4.3.1 on windows (reference SPL-48993). This is caused by the fact that the timezone of your splunk user (probably admin) has been changed in his profile. It should be fixed in 4.3.2, but until version 4.3.2 is shipped, the workarounds are : change the timezone in the user account to "default system timezone" or add the license-master manually in the file $SPLUNK_HOME/etc/system/local/server.conf [license] active_group = Enterprise master_uri = https://server:8089 ... View more

Hi Chris The root cause here is that in the app "asset_discovery" , the eventtype in this case is referencing a savedsearch. But in a distributed search setting, splunk doesn't replicate savedsearches.conf from the search-head to the peers. The problem is that the app is not using a conventional definition for the eventtypes. that is not supported. Workarounds : install the app in the search-peers change the bundle replication whitelist to add the savedsearches.conf ( will be more costly for all your apps / searches ) ask the author of the app to update his app to be compatible with distributed search. wait for an enhancement in splunk to allow this. ... View more

Update : Details are on this wiki page : http://wiki.splunk.com/Community:TroubleshootingIndexedDataVolume remark : License_usage.log is available in the Splunk license master instance only. A license master logs indexed events volume every minute by the information the slaves send to the master. A slave maintains a table of how much you've indexed on a slave in chunks of time. Typically that chunk of time is 1 minute, but the chunk may grow if the slave cannot contact the master -- Splunk only resets the chunk when the table is sent to the master. The table is of src,srctype,host tuples… if that table grows to exceed 1000 entries, then Splunk squashes the host/source keys. So, if you have more than 1000 different tuple entries, you find no value for h(ost) and s(ource) fields. Splunk never suppresses st(sourcetype) in the log. ... View more

Details are on this wiki page : http://wiki.splunk.com/Community:TroubleshootingIndexedDataVolume remark : License_usage.log is available in the Splunk license master instance only. A license master logs indexed events volume every minute by the information the slaves send to the master. A slave maintains a table of how much you've indexed on a slave in chunks of time. Typically that chunk of time is 1 minute, but the chunk may grow if the slave cannot contact the master -- Splunk only resets the chunk when the table is sent to the master. The table is of src,srctype,host tuples… if that table grows to exceed 1000 entries, then Splunk squashes the host/source keys.0 So, if you have more than 1000 different tuple entries, you find no value for h(ost) and s(ource) fields. Splunk never suppresses st(sourcetype) in the log. ... View more

or in an app if you deploy your configuration in apps. $SPLUNK_HOME/etc/apps/ /local/ ... View more

FYI, in splunk 4.3.1, to help you identify this issue, a new warning message has been added in splunkd.log to report a slow dns resolution. ... View more

You may have identified the issue, the reverse DNS is taking too long. Do you know if your DNS is quick to resolve an Ip/hostname, can it resolve all of your forwarders, are they defined in your dns server, are you under Windows with netbios reslution turned on ? A first workaround to reduce the resolution time, by example by populating a local host list file, or changing the DNS settings. A second workaround is to disable the dns resolution in splunk for your splunktcp inputs. the resolution is in general useless, because the host field is already populated by the forwarder. the only consequence is that in splunk metrics, the source field will be the ip instead of the hostname. if needed you do this also for UDP/TCP syslog inputs, but this will change the format of your syslog logs. To proceed, edit the inputs.conf and add the parameter connection-host = none see http://docs.splunk.com/Documentation/Splunk/4.3.1/admin/Inputsconf [splunktcp://9997] connection_host = none ... View more

Hum, not the best idea, removing raw data means that you will not be able to access the data after, therefore those cold buckets will be useless. if you really want to delete cold buckets, then setup a retention policy (on total index size or on time retention.) ... View more

The way to reduce your indexed volume and to get rid of useless events, is to setup filtering on the indexer. Usually based on sourcetype, and a regex mach on the events. See example in the docs http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Routeandfilterdatad#Filter_event_data_and_send_to_queues ... View more

Concerns current version of Splunk Storm too. ... View more

Install sos app on the Search-head, enable the ps_sos.sh scritped input. It will collect measures cpu/memory per splunk process, you may be able to see the detail. (if this is splunkd, splunkweb, or user searches) http://splunk-base.splunk.com/apps/29008/sos-splunk-on-splunk ... View more

try 0 23 * * 0 the 5th part is for the day of the week (0=sunday) keep the third with a * it will be controlled by the condition on the day of week. * * * * * command to be executed | | | | | | | | | +----- day of week (0 - 6) (Sunday=0) | | | +------- month (1 - 12) | | +--------- day of month (1 - 31) | +----------- hour (0 - 23) +------------- min (0 - 59) ... View more