About mas

mas · ‎04-08-2020

Hello, some updates. I focused on a short time window for a specific dataset and I found out that accelerated searches ("tstats", "from datamodel" and "datamodel") return 4 events. On the other hand, raw searches, built both from datamodel definition and using "| datamodel flat_string", return 11 events in the same time window. The really strange thing is that the acceleration search, executed on the same time window, returns 11 events. I retrieved the acceleration search with "| datamodel acceleration_search_string" and I executed it with a filter on "nodename=". It looks like, in some way, summary indexes do not store all the expected data. Acceleration searches run every 5 minutes. Sometimes they are skipped due to concurrency limits, but their execution is later recovered. Any clues or ideas?

mas · ‎04-07-2020

Hello everybody, I see a strange behaviour with data model acceleration. I have a data model accelerated over 3 months. According to internal logs, scheduled acceleration searches are not skipped and they complete providing results. However if I run a tstats search over last month with “summariesonly=true”, I do not get any values back; if I run the same tstats search with “summariesonly=false”, I do get expected results. Again, if I run the tstats search over the last 90 days with "summariesonly=true", I get some values back. Have you ever faced a similar situation? Could this depend upon the small number of events, thus upon buckets not rolled yet? Please not that this does not look like a generic "recent data not yet summarised" issue, because: acceleration searches complete with success every 5 minutes; data model summary is 100% built; I am missing data at least from the last month. Thank you for your support!

mas · ‎01-31-2020

Hi! Anyone experienced with the new AddOn Builder (3.0.1)? I would like to implement some logic before saving an "add-on setup parameter" or a "data input parameter". To be more precise, I would like to manipulate string values, concatenating them and encoding them with Base64. I spent some time doing reverse-engineering of Python code, but it looks like parameters are saved according to some generic logic and it is not possible to override it for specific parameters. Which is the right way to do this? Where should I place my code, so that it survives to future changed made using the Add On Builder?

mas · ‎07-18-2019

Hi all, according to official documentation, all indexer cluster nodes (master node, peer nodes, and search heads) must run on the same operating system (official documentation). It's not clear to me if this mean the same operating system type (e.g. SLES) or the same operating system type in the same version (e.g. SLES v12.3). We are going to "upgrade" the cluster master and some search heads to a newer operating system version, creating new machines that will substitute the existing ones. We would like to move from SLES v11.4 to SLES v15.1, but indexers are running on SLES v12.3. Mixing SLES v.12.3 (on indexers) and SLES v.15.1 (on cluster master and search head) could cause issues? In any case, we would install the same Splunk version on all cluster nodes.

mas · ‎10-08-2018

Hi, One of my customers asked to add a field to the "Edit notable event" popup form in Splunk ES 5.1.1. To be more precise, customer wants an "Incident category" field, that must be populated by Security Analyst, picking up a choice from a restricted set of values (something very similar to "Status" field). I know that I can add a "table attribute" using incident review customization page, but: - incident category is not associated to a field in originating event (it is classified by Security Analyst); - incident category values must be selected using a drop-down, from a restricted set of values. Do you have any suggestions? Thank you!

mas · ‎02-14-2018

Your solution works, with a bit of hacking on checkpoint files. Thank you!

mas · ‎11-28-2017

Hi, I am using DB Connect 3.1.1 to get data from a Microsoft SQL Server 2014 database. I am required to collect the results of the invocation of some stored procedures. According to the documentation, dbxquery command supports stored procedures in databases beginning with version 3.0.0. However I am not able to configure a DB input based on a stored procedure. In fact db_inputs.conf.spec only admits a "query" parameter (while dbxquery command allows the use of a "procedure" parameter"). If i try to "trick" the "Edit Input" GUI, using a SQL statement that contains "exec , I am able to retrieve results, but columns are returned in alphabetic order and "Rising column" / "Timestamp column" dropdowns do not contain selectable values. Finally, if I put the same "exec " in db_inputs.conf, I get some errors due to missing checkpoint file (I created it manually to solve the issue) and missing metadata files (no idea where they should be and I stopped, because I don't want to do heavy "hacking" of the app). Any suggestions on how to create a DB input based on a stored procedure?

mas · ‎03-20-2017

Thank you cmutt78, your solution is working. By the way: there is some additional, useful information at this link: https://wiki.splunk.com/Community:Run_multiple_Splunks_on_one_machine (this is specific for multiple Splunk instances on the same box).

mas · ‎12-01-2016

Hello guys, We are going to install two Heavy Forwarders on Windows 2012 R2 servers. The remaining instances of Splunk, which build up our distributed architecture, are running on SLES. As usual, according to best practices, I was trying to distribute our "master" splunk.secret file to new Heavy Forwarders hosted on Windows servers. I tried to install Splunk using the following command line: msiexec.exe /i splunk-<...>-x64-release.msi AGREETOLICENSE=Yes WEB_PORT= DEPLOYMENT_SERVER="" LAUNCHSPLUNK=0 INSTALL_SHORTCUT=0 As expected the "splunkd" service did not start when installation finished, but unluckily a new splunk.secret was automatically created and contents where encrypted using it. So I tried an interactive installation with only the "LAUNCHSPLUNK=0" flag and I monitored the file system: I noticed that the splunk.secret and the encrypted files are created at the same exact time, before the service is started. QUESTION: is it possible to install Splunk on Windows without the creation of a new splunk.secret and the subsequent encryption of data with it, in the same way it is possible in Linux? Thank you!

mas · ‎09-01-2015

Workaround seems to work correctly.

mas · ‎09-01-2015

Actually the workaround seems to work correctly: in these weeks I had not any problem with the extractions. However I submitted a "bug report" to Splunk. I think this question can be closed, considering the workaround as solution.

mas · ‎08-19-2015

Hi, I have found a strange behaviour in the Splunk Add-on for Windows: it contains some transformations, used to extract CIM-compliant fields, which are not correctly processed. After some analysis, I found out that all the transforms which include a REGEX ending with an escaped "backslask" are not processed correctly. For instance, this transformation is not processed at search time: [Security_ID_as_dest_nt_domain] SOURCE_KEY = Security_ID REGEX = (.+)\\ FORMAT = dest_nt_domain::"$1" In addition, when I try to edit these transformations using the GUI, I see that the existing REGEX is read by Splunk as: (.+)\\FORMAT = dest_nt_domain::"$1" In my opinion, something is going wrong when Splunk is parsing a regex ending with an escaped backslash. Has anybody incurred the same problem? I am using Splunk 6.2.0 and Splunk_TA_windows 4.7.4 (but I see the same regex in transforms.conf of Splunk_TA_windows 4.7.5). At present, as a workaround, I have created a "transforms.conf" in Splunk_TA_windows/local , overriding the transforms which use regular expressions ending with an escaped backslash. I rewrite the REGEX enclosing the ending backslashes in square brackets. For instance, the above mentioned transformation becomes: [Security_ID_as_dest_nt_domain] SOURCE_KEY = Security_ID REGEX = (.+)[\\] FORMAT = dest_nt_domain::"$1"

mas · ‎02-02-2015

Thank you. It would be nice to be able to check the cluster status before starting the backup job, raising a warning if it is not consistent. I think the only way to automate this check would be to run "bin/splunk show cluster-status --verbose" and check for "Replication factor met" and "Search factor met", but this command is available only on master node, while the check should be executed by the backup agent running on one of the peers. In addition, there would be an authentication request from Splunk daemon. As a result, I think it is not possible to automate this check in a simple way.

mas · ‎02-02-2015

Hi, I suggest you to rise the logging level for TcpOutputProc. You can do this using web interface or editing "log.cfg" directly (in this case, changes will persist even when you restart Splunk). More details available here. Could you send some logs, when you have changed this configuration? In the past I saw some similar connection problems due to SSL misconfigurations in Splunk-to-Splunk connections (i.e. forwarder uses SSL, while indexer does not, or vice-versa). However I must say that it does not look your case. Could you please try to run the following commands and send resulting output? On the Universal Forwarder: splunk cmd btool server list --debug splunk cmd btool outputs list --debug On the Indexer: splunk cmd btool server list --debug splunk cmd btool inputs list --debug

mas · ‎01-30-2015

Hi! Some questions to start troubleshooting. In the log of the Universal Forwarder ($SPLUNK_HOME/var/logs/splunk/splunkd.log), do you see any TcpOut errors occurring periodically, with the same frequency connection attempts are made? Have you configured Splunk to use SSL connections for data forwarding/receiving (on the Universal Forwarder/Indexer)? When you say "On the Splunk server side, receiving port 9998 is enabled", do you mean that TCP port is open? Or do you mean you have configured Splunk receiving on port 9998?

mas · ‎01-30-2015

Hi, we have a multisite indexers cluster with two nodes and replication factor = 2. All instances are running Splunk 6.2. In the official documentation we read that it is not possible to back up just the data on a single node, since there's no certainty that a single node contains all the data in the cluster. I suspect that this is not true in our scenario, where all data are replicated to both nodes (replication factor = total number of peers). Am I right? Can we backup data from one node (snapshot backup) and be sure that all data has been saved?

mas · ‎01-30-2015

Thank you very much for your answer: I think that this new capability will be really useful in environments with strict security requirements. By the way: in the meantime, four our specific needs, we moved to "DB Connect" app as a workaround. In fact our requirement was directly related to low-privileges access to a Microsoft SQL Server instance from the Universal Forwarder installed on the same server. A Universal Forwarder was preferred, compared to direct connections to DB, due to the possibility to encrypt traffic on the network and to the need to reduce firewall ports openings. Just one annotation: in security-hardened environments, it is common to remove system privileges from default groups. As a result, adding the domain account to "Performance Log Users" and "Event Log Readers" groups could not lead to the expected result (that is, forwarder could be unable to read performance logs and event logs). In this scenario, Splunk administrator should be warned to assign these permissions explicitly, for instance by means of Group Policy Objects.

mas · ‎11-18-2013

Unluckily I have now way to identify the error that occurred immediately before the TcpChannel messages, because these are generated too fast and the splunkd.log files are flooded with them. The only error we can see in the splunkd.log and its associated rotation files is the TcpChannell message, repeated indefinitely.

mas · ‎11-18-2013

Hi, after the upgrade to Splunk 6 from Splunk 4.3.3, we have serious problems with our single-server instance (Windows Server 2008 R2). In fact during the night the CPU usage of "splunkd" processes reaches 100% and hangs at this level, making the server unreachable via WEB interface, but often also via RDP connections. The only solution is to terminate the splunkd process and then restart it. Having a look at splunkd.log, we see a continuous flooding of these errors (over 35.000 errors per second): ERROR TcpChannel - Error trying to begin socket accept: An invalid argument was supplied. These errors disappear after splunkd service has been restarted. We guess there is some problem with inbound connections from Universal Forwarders (that have not been upgraded yet), but we have no clue to confirm this diagnosis. System event logs do not report any warning or error. Any suggestions would be really appreciated. Regards.

mas · ‎08-19-2013

I found the problem. The same day of the update, a configuration change had been made in props.conf (under ./etc/system/local). A stanza had been created to match data sent by three specific hosts (just say: serverA, serverB and serverC). There was a typo in this stanza name! It had been defined as: [host::serverA|host::serverB||host::serverC] instead of [host::serverA|host::serverB|host::serverC] The double "||" caused a "match-all" situation. In addition, this stanza has a conditional redirect of all the events, with the exception of the required ones, to the null queue. All the syslog events, as a result, were discarded!

Posts	24
Solutions	2
Karma Given	0
Karma Received	10
Member Since	‎05-10-2012

Online Status	Offline
Date Last Visited	‎06-05-2020 02:04 AM

Accelerated data model returning partial results w...

Implement logic when saving parameters using Add O...

Indexer cluster: same operating system, but with d...

In Splunk Enterprise Security, how do I add a New ...

Splunk DB Connect 3.1.1 - Stored procedure as inpu...

How to distribute splunk.secret to Windows Heavy F...

Possible bug: some field extractions of Splunk Add...

Backup of indexed data in cluster with two nodes a...

CPU usage of splunkd processes hangs to 100% after...

Syslog data source and Splunk 5.0.3 on Windows 200...

Re: Accelerated data model returning partial resul...

Accelerated data model returning partial results w...

Implement logic when saving parameters using Add O...

Indexer cluster: same operating system, but with d...

In Splunk Enterprise Security, how do I add a New ...

Re: Splunk DB Connect 3.1.1 - Stored procedure as ...

Splunk DB Connect 3.1.1 - Stored procedure as inpu...

Re: How to distribute splunk.secret to Windows Hea...

How to distribute splunk.secret to Windows Heavy F...

Re: Possible bug: some field extractions of Splunk...

Re: Possible bug: some field extractions of Splunk...

Possible bug: some field extractions of Splunk Add...

Re: Backup of indexed data in cluster with two nod...

Re: universal forwarder deployment on a windows fa...

Re: universal forwarder deployment on a windows fa...

Backup of indexed data in cluster with two nodes a...

Re: Running Universal Forwarder with non-administr...

Re: CPU usage of splunkd processes hangs to 100% a...

CPU usage of splunkd processes hangs to 100% after...

Re: Syslog data source and Splunk 5.0.3 on Windows...