We have nine sites in a multi-site cluster with indexers at each site ranging from three to 15 servers. Each site's indexers are all on the same vlan and ip subnet for for their region. I have a need to expand one of the sites with more indexers but the vlan has run out of IP addresses. Is it possible to just create a new vlan to use a different IP subnet range and add these new indexers to the previously configured site? For example site2's indexers are in vlan 2 on ip subnet range 10.1.1.0/24. Can I add 6 new indexers to site2 with those new servers in vlan 3 on ip subnet range 11.1.1.0/24? I looked over the documentation and didn't see a requirement indexers in a site for a multi-site cluster need to all be on the same vlan/ip subnet range but wanted to check if this is a legit configuration from real users in the community. Any pro's and con's? We currently have two independent search heads but are going to a search head cluster later this year. thank you ... View more

I need to determine the significance of these errors before giving the green light to upgrade production. These are all related to after upgrading enterprise from 7.0.5 to 7.3.3 and Splunk Enterprise security from 5.0.1 to 5.3.1. ERROR 2019-12-10 10:01:52.639 security SearchParser Missing a search command before ''. Error at position '2' of search query '| * inputlookup append=T http_intel * where * * * '. ERROR 2019-12-10 10:01:52.649 security SearchParser Missing a search command before ''. Error at position '2' of search query '| * inputlookup append=T email_intel * where * * '. ERROR 2019-12-10 10:01:52.670 security SearchParser Missing a search command before ''. Error at position '2' of search query '| * inputlookup append=T service_intel * where * '. ERROR 2019-12-10 10:01:52.679 security SearchParser Missing a search command before ''. Error at position '2' of search query '| * inputlookup append=T registry_intel * where * '. ERROR 2019-12-10 10:01:52.689 security SearchParser Missing a search command before ''. Error at position '2' of search query '| * inputlookup append=T process_intel * where * *'. ERROR 2019-12-10 10:01:52.708 security01-dev SearchParser Missing a search command before ''. Error at position '2' of search query '| * inputlookup append=T user_intel * where * * * '. ERROR 2019-12-10 10:01:52.718 security01-dev SearchParser Missing a search command before '*'. Error at position '2' of search query '| * inputlookup append=T certificate_intel * where'. If I run a grep on all files with these variable such as http_intel, email_intel, service_intel, the results show them in ESS DA dashboards. We never saw this error prior to upgrading to 7.3.3 Below is a sample, all others are about identical. ERROR 2019-12-10 04:01:39.131 security SearchParser Missing a search command before '*'. Error at position '2' of search query '| * inputlookup append=T http_intel * where * * * grep on http_intel found the below dashboard search DA-ESS-ThreatIntelligence/default/data/ui/views/threat_artifacts.xml:432: | $tab_threat$ inputlookup append=T process_intel $max$ where * $network_filter$ $file_filter$ $registry_filter$ $service_filter$ $user_filter$ $process_filter$ $certificate_filter$ $email_filter$ | set_threat_collection_name("process_intel") | eval ip=mvappend(src, dest), domain=mvappend(src, dest) | inputlookup append=T certificate_intel $max$ where * $network_filter$ $file_filter$ $registry_filter$ $service_filter$ $user_filter$ $process_filter$ $certificate_filter$ $email_filter$ | set_threat_collection_name("certificate_intel") | mvexpand certificate_serial | get_certificate_serial | eventstats values(certificate_serial) as certificate_serial,values(certificate_serial_clean) as certificate_serial_clean,values(certificate_serial_dec) as certificate_serial_dec by _key | dedup _key,threat_collection | inputlookup append=T email_intel $max$ where * $network_filter$ $file_filter$ $registry_filter$ $service_filter$ $user_filter$ $process_filter$ $certificate_filter$ $email_filter$ | set_threat_collection_name("email_intel") | inputlookup append=T ip_intel $max$ where * $network_filter$ $file_filter$ $registry_filter$ $service_filter$ $user_filter$ $process_filter$ $certificate_filter$ $email_filter$ | set_threat_collection_name("ip_intel") | inputlookup append=T http_intel $max$ where * $network_filter$ $file_filter$ $registry_filter$ $service_filter$ $user_filter$ $process_filter$ $certificate_filter$ $email_filter$ | fillnull value=0 updated,disabled | set_threat_collection_name("http_intel") | get_threat_attribution(threat_key) | search $threat_id_filter$ | eval ip=coalesce(embedded_ip,ip), domain=coalesce(embedded_domain,domain), file_hash=coalesce(certificate_file_hash,file_hash), src_user=coalesce(certificate_subject_email,src_user), actual_src_user=coalesce(certificate_issuer_email,actual_src_user), file_name=coalesce(process_file_name,file_name), file_path=coalesce(process_file_path,file_path) | mvappend_field(url, http_referrer) | fillnull value="" threat_collection, source_type, threat_group, threat_category, malware_alias| stats dc(ip) as ip_count, dc(domain) as domain_count, dc(url) as url_count, dc(http_user_agent) as http_user_agent_count, dc(header) as header_count by threat_collection, source_type, threat_group, threat_category, malware_alias | addtotals fieldname=http_count http_user_agent_count, header_count | addtotals fieldname=total ip_count, domain_count, url_count, http_count | where total > 0 | fields threat_collection, source_type, ip_count, domain_count, url_count, http_count, total, threat_group, threat_category, malware_alias | sort - total ... View more

After upgrading to 7.3.3 from 7.0.5 these two log ERRORs are new ERROR 2019-12-10 08:01:19.755 security TsidxStats Missing search clause after 'WHERE' keyword 1 ERROR 2019-12-10 08:01:46.309 security TsidxStats Wildcards (*) are not supported in aggregate fields 1 I found a similar log message where it mentions this is a bug. https://answers.splunk.com/answers/593866/how-to-resolve-this-error-error-in-tsidxstats-wher-1.html Has anyone seen these two log messages? I'm trying to gauge the significance before upgrading our production environment. ... View more

Upgraded from 7.0.5 to 7.3.3 and noticed splunkd Datamodel log ERRORs for removed macros ERROR DataModelObject Failed to parse baseSearch. err=Error in 'SearchParser': The search specifies a macro 'search_activity' that cannot be found. Reasons include: the macro name is misspelled, you do not have "read" permission for the macro, or the macro has not been shared with this application. Click Settings, Advanced search, Search Macros to view macro information., object=Search_Activity, baseSearch= search_activity Support recommended just delete the .json file but how do I know I'm not breaking anything? So...this brought up the below problem Problem: The macro 'search_activity' has been removed in 7.3.3 yet the datamodel schema .json files in the local/data/models directory still references this macro and the splunkd logs are showing error messages the macro no longer exists. /opt/splunk/etc/apps/Splunk_SA_CIM/local/data/models/Splunk_Audit.json "calculations": [], "constraints": [], "lineage": "Search_Activity", <<does not exist in default CIM 4.13 "baseSearch": " search_activity " <<does not exist in default CIM 4.13 another flavor of this problem is the change of .json files (datamodel schema) between CIM versions. I have the local/Authentication.json from 4.11 below that is totally different from 4.13's default Authentication.json 4.11 below local/ /opt/splunk/etc/apps/Splunk_SA_CIM/local/data/models/Authentication.json | less { "modelName": "Authentication", "displayName": "Authentication", "description": "Authentication Data Model", "objectSummary": { "Event-Based": 10, "Transaction-Based": 0, "Search-Based": 0 }, "objects": [ { "objectName": "Authentication", vs 4.13 default/Authentication.json -bash-4.2$ cat /opt/splunk/etc/apps/Splunk_SA_CIM/default/data/models/Authentication.json | less { "modelName": "Authentication", "displayName": "Authentication", "description": "Authentication Data Model", "editable": false, "objects": [ { "comment": { "tags": [ "authentication" ] }, "objectName": "Authentication", Question: How are customers supposed to upgrade to the new CIM versions and use the new default/.json files if local is overriding those changes? I do not see anywhere how these local json files were created in the first place by us users. Do we reverse engineer how these local/.jsons were created, delete the local/*.json files and try to create them again in the gui on the newer CIM version? This is not clear at all and I would appreciate any guidance for best practices. I have no idea what will break if I just remove all my local *.json files. I did not see this issue mentioned anywhere in the upgrade documentation. I assume this is common. I did look to see what changes are done to files in the Managed Apps, CIM->Setup and here we changed tags and indexes which touches the local macros.conf and datamodels.conf and the Settings->Datamodels->edit acceleration touches the local datamodels.conf. I see nothing that creates the local .json files. thank you ... View more

On 7.0.5 with our Search head using Enterprise Security we were able to run Search and Reporting searches, |tstats searches for our ESS correlation rules and | datamodel searches such as "|datamodel Authentication Failed_Authentication search | search index=os sourcetype=linux_secure" with no issues. I use the |datamodel searches to make sure the datamodel is picking up the fields in my logs before writing correlation rules. All worked until we upgraded to 7.3.2. Now normal search and reporting still works, |tstats searches for correlation rules still work but |datamodel searches do not find any events. It says to "No results found. Try expanding the time range." However, If I remove the "index=os" from the same datamodel search such as the the below |datamodel Authentication Failed_Authentication search | search sourcetype=linux_secure Splunk results return but the only fields parsed are those from the Authentication datamodel, all the other fields you would normally see under a "Search and Reporting" search are gone such as index, user etc. Issue has been brought to Splunk support but no comment. I was curious if anyone else has seen this issue. We did rebuild our datamodels but no difference. ... View more

I'm not sure why the app makers just don't change the name of the app to TA-Sudo so the regex for importing apps in ESS works right from the get go. They mention to change the regex for TA_ but perhaps that imports unwanted apps into ESS that you may not want imported. It just seems changing ESS regex is more dramatic than just changing the app name by the makers to work with ESS. ... View more

I have Indexers in a cluster running Splunk_TA_nix. I'm monitoring /var/log in inputs.conf. I can see the log events from the search head with a splunk_server from a different Indexer in the cluster. Two questions 1) How did the /var/log/messages, as an example, get indexed? Did it get indexed locally, and if so, how did it know to do that? Or did the events get forwarded to other indexers in the cluster like how our heavy forwarders use Indexer-discovery by contacting the Cluster master for the list of indexers? I ask because I do not see any outputs.conf being configured on Indexers showing any auto-discovery. The cluster master settings are only in the server.conf file. I do not see how these local OS logs are being indexed and it's bothering me. 2) Can I assume the search results showing the /var/log/messages from host1 being seen in the results as splunk_server=host2 is due to replication or is it from host1 forwarding to host2 for indexing? thanks ... View more

Running syslog-ng with a HF. Logrotate runs hourly. 16 or so different web proxies are sending logs to the syslog-ng server with the HF. Sometimes 1 out of the 16 proxy log sources are no longer getting read by the HF even though the proxy log file exists in syslog-ng and can be read. At the top of the hour it fixes itself and the HF reads the file but i'm out of logs for an hour for correlation rules. DEBUG was enabled and below shows the log right at the time of the last log event seen in Splunk 01-25-2019 22:00:05.997 +0000 DEBUG TailingProcessor - Defering file=/var/syslog/proxy/192.168.251.141/proxy.log unsafe as it does not exist anymore, scheduling a oneshot timeout instead. ./splunk list inputstatus | grep -A4 proxy | grep -A4 192.168.251.141 /var/syslog/proxy/192.168.251.141 parent = /var/syslog/proxy//.log type = directory <<<<<<< >>>>>>> /var/syslog/proxy/192.168.251.141/proxy.log file position = 30610690 file size = 30610690 parent = /var/syslog/proxy//.log percent = 100.00 I read logrotate should not be used with syslog-ng. Anyone ever see this message? ... View more