Unfortunately our proxy data does not have user information. However I do have access to AV data that is able to map client IP to user information. Via the "Lookup Definitions" link in the Splunk Manager I can setup Max and Min Offset for my "enrichment data". I see that these settings would used if my "enrichment" data is in the future. Unfortunately my enrichment data is usually one day behind. This is causing issues with the correct encrichment data being applied to the events. Config below: [av_lookup] filename = av_lookup.csv time_field = savreportcheckin lookup_table = av_lookup ip_address AS c_ip OUTPUTNEW clientuser computer savreportcheckin Enrichment data below: clientuser,ip_address,savreportcheckin,computer u000000,10.0.0.0,2010-07-26 22:24:00,WP103702A740532 z000000,10.0.0.0,2010-07-27 22:23:00,WP103702A740532 If I search on this event data: 2010-07-26 22:55:09 3 10.0.0.0 200 TCP_RESCAN_HIT 2597 851 GET http www.newyorklife.com 80 - - 206.210.18.92 - "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" Financial%20Services - 127.0.0.1 - 2010-07-26 22:55:09 1 10.0.0.0 200 TCP_HIT 7268 866 GET http www.newyorklife.com 80 - - 206.210.18.92 - "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" Financial%20Services - 127.0.0.1 - The z00000 user ID is returned. This is the incorrect user id since the event occured on 07-26 not 07-27 and the person updated there AV sigs approximately 30 mins before this event happened. Is there a config within lookup definitions to only match "enrichment" data to events that occured within so many seconds either before or after the actual event occured ? Thanks for the help, Scott ... View more

We are currently performing a POC using Splunk 4.1.3 to index Blue Coat proxy data. Our test Splunk license is for 200gig a day. I have indexed around 800gig worth of raw data on our POC server. The Splunk documentation @ http://www.splunk.com/base/Documentation/latest/Installation/HowHowmuchspaceyouwillneed gives some commands to run in order to get a idea of how much storage you will need. I ran the commands and see the following numbers, I am only using the default DB: [root@ssdatacrusher2 db]# du -shc hot_v*/rawdata 3.2G hot_v1_22/rawdata 31M hot_v1_25/rawdata 3.9G hot_v1_35/rawdata 2.7G hot_v1_38/rawdata 9.8G total [root@ssdatacrusher2 db]# du -ch hot_v* 4.0K hot_v1_22/splunk_optimize_logs 4.0K hot_v1_22/rawdata/.compressedAddresses 3.2G hot_v1_22/rawdata 8.0G hot_v1_22 0 hot_v1_22.sentinel 4.0K hot_v1_25/splunk_optimize_logs 4.0K hot_v1_25/rawdata/.compressedAddresses 31M hot_v1_25/rawdata 62M hot_v1_25 0 hot_v1_25.sentinel 4.0K hot_v1_35/splunk_optimize_logs 4.0K hot_v1_35/rawdata/.compressedAddresses 3.9G hot_v1_35/rawdata 9.5G hot_v1_35 0 hot_v1_35.sentinel 4.0K hot_v1_38/splunk_optimize_logs 4.0K hot_v1_38/rawdata/.compressedAddresses 2.8G hot_v1_38/rawdata 7.3G hot_v1_38 0 hot_v1_38.sentinel 25G total I however ran a du -h on the $SPLUNK_HOME/var directory and am seeing around 400gig worth of disk usage: 406G ./var 406G . [root@ssdatacrusher2 splunk]# pwd /opt/splunk I am seeing most of the 400gig disk usage being used by directories that end with rawdata. When the data is rolled from hot to warm will these directories be deleted and cause my disk usage to do down or stay around the same since I will still be indexing more raw data ? ... View more