OK one of our devs discovered a weird bug where if a lookup is being performed on a CSV where the field to match contained spaces. The results are consistently weird between Linux and Windows. It doesn't matter if the CSV lookup is Unix or Windows format (ie CRLF or just LF newlines) Placing the lookup field in quotes makes no difference. I have a sample app here which demonstrates the bug using internal Splunk data. http://www.phoenixdigital.com/splunk/lookup_bug.spl The screenshot shows the bug in action. (see attached screenshot) You can see in this example the group=pipeline data does not appear in the results on the left however it does on the data on the right when you change group_is_it_searchable=no to group_is_it_searchable=*no I will put the major components here for clarity though. props.conf [splunkd] LOOKUP-1-first = group_extend group OUTPUT group_spaced, group_bug_prediction LOOKUP-2-second = spacing_issue group_spaced OUTPUT group_is_it_searchable transforms.conf [group_extend] filename = group_extend.csv [spacing_issue] filename = spacing_issue.csv group_extend.csv (This lookup creates a field 'group_spaced' which will be used for the next lookup) group,group_spaced,group_bug_prediction pipeline,pipe line,this will break queue,queue,this will work per_host_thruput,per host thruput,this will break tpool,tpool,this will work deploy-connections,deploy connections,this will break map,map,this will work per_source_thruput,per source thruput,this will break spacing_issue,csv (The second lookup which provides inconsistent results) group_spaced,group_is_it_searchable queue,yes per host thruput,no tpool,yes pipe line,no deploy connections,no map,yes per source thruput,no ... View more

So it appears that the built-in tagging and field enrichment for the Splunk App for Enterprise Security is poorly configured. For the Change Analysis CIM, I was pleased to see Windows events being tagged on ES http://docs.splunk.com/Documentation/CIM/4.2.0/User/ChangeAnalysis However, the field enrichment is totally off. Pay particular attention to user - The user or entity performing the change (can be UID or PID). object - Name of the affected object on the resource (such as a router interface, user account, or server volume). object_category - Generic name for the class of the updated resource object. Expected values may be specific to an App. src_user - The resource where the change was originated. May be aliased from more specific fields, such as src_host, src_ip, or src_name. When this Windows event arrives where: ad_jus = the admin user creating the user JSMITH = the user being created 09/10/2015 12:09:45 PM LogName=Security SourceName=Microsoft Windows security auditing. EventCode=4720 EventType=0 Type=Information ComputerName=ADSVR1.production.prod TaskCategory=User Account Management OpCode=Info RecordNumber=2472352 Keywords=Audit Success Message=A user account was created. Subject: Security ID: PROD\ad_jus Account Name: ad_jus Account Domain: PRODUCTION Logon ID: 0x3ff7132B5B New Account: Security ID: PROD\JSMITH Account Name: JSMITH Account Domain: PRODUCTION Attributes: SAM Account Name: JSMITH Display Name: John Smith User Principal Name: JSMITH@production.prod Home Directory: \\production.prod\fsroot\UserData\JSMITH Home Drive: H: Script Path: - Profile Path: - User Workstations: - Password Last Set: <never> Account Expires: <never> Primary Group ID: 513 Allowed To Delegate To: - Old UAC Value: 0x0 New UAC Value: 0x11 User Account Control: Account Disabled 'Normal Account' - Enabled User Parameters: - SID History: - Logon Hours: <value not set> Additional Information: Privileges In Enterprise Security the fields are extracted as follows: user = JSMITH object = WinEventLog:Security object_category = user src_user = ad_jus Seems to me, based on the CIM, it should be: user = ad_jus object = JSMITH object_category = user src_user = ad_jus Ideally I could try having custom local props and transforms in Splunk_TA_windows to fix this issue, but seeing as Splunk ES is a paid product, I would have thought it should probably be fixed for all customers. I am not sure if this affects other EventCodes related to Windows user management, but it likely does. I am posting this here so support can have a reference to the issue. If anyone knows of a good quick solution please let me know. Using Splunk TA Windows Build 261729 Splunk ES version 3.3.0 ... View more

Hi All, Just wanted to see what ways you have done to convince administrators to give you the recommended Splunk CPUs when installed in a virtual environment. I have encountered this 3 times in as many weeks when visiting client sites. One recent site the admins only gave the search heads one CPU inititally which was just ridiculous. We convinced them to up it to 4 but it is still in my opinion woefully inadequate. The administrators stated we will watch the servers and if we see high usage we will increase the cores. I explained to them that on startup Splunk will check how many cores it has available and throttle itself accordingly based on settings in limits.conf. The other argument is sure Splunk may run with low CPU usage 90% of the time but the other 10% of the time it needs that power for complex searches and analysis. Short of artificially increasing usage until we get the cores required what methods have others used to convince VM administrators to give your Splunk installation the recommended CPU and memory sizes? ... View more