Ran into the same issue with one of my customers. We found that removing the file "custom.xml" located in default/data/ui/nav in the Okta add-on fixed the issue, and still let us use the search-time parsing in ES for Okta events. I'm not sure why custom.xml is there, since it is identical to default.xml in the same directory. ... View more

Does your data contain timestamps? I don't see any in your sample logs above, but I'm curious is there are timestamps in the missing portions of the data. ... View more

Just to make sure we are checking everything, what does your Splunk environment look like? If you're running distributed, are you forwarding your search head logs to your indexers? ... View more

I don't know of an easy way. Your best bet is to you strptime and/or strftime to create a calculated field based on the additional timestamps in the data, and create a dashboard where users can input their own timestamps (for earliest and latest) to compare to the fields you want to search on. It's not going to be easy to work out, but it should be possible using the simple-xml structure available in dashboards. ... View more

I would set up your inputs and props with the following configuration (replacing the ## variables with what matches your case): inputs.conf: [monitor://#path to your files#] sourcetype = #your sourcetype# props.conf: MAX_TIMESTAMP_LOOKAHEAD = #timestamp character count# LINE_BREAKER = SaveRetailItemSalesLocation\>([\r\n\s]+) TIME_FORMAT = #timestamp Format# TIME_PREFIX = TransactionDate\> SHOULD_LINEMERGE = false TRUNCATE = #max size of your events# KV_MODE = xml If you put the props.conf on your search heads, indexers, and forwarders, you should get events that use the TransactionDate as their timestamp, with all their fields auto-extracted in spath format. I made assumptions on the timestamp field, which can be adjusted if you need a different field to act as the timestamp. ... View more

Do the events appear complete when you search for them via "host"? Meaning, the JSON does not appear truncated in the event viewer. I would imagine that you are running up against the default TRUNCATE option for your sourcetype (in props.conf), which by default is set to 10000 bytes. I would try setting TRUNCATE for your sourcetype higher, and then coming back here if that does not work. ... View more

Which servers in your environment does the message match? As in, do the IPs match your search heads, indexers, forwarders, or none of the above. ... View more

Just to clarify: your XML files should be read as a single event per file? Can you post an example of the XML structure? ... View more

I would need to see your inputs.conf, props.conf, and transforms.conf for your particular input, but my first guess would be to investigate your settings in props.conf for your sourcetype. ... View more

Can you provide a few more details. It's not clear what you want to match. Your regex seems to match everything from the first two open tags to the first close tag of your XML. ... View more

What is the end-goal for this? With Pivot reports, you have the option of saving them with a time range picker, which should provide multiple time-filters for the report. ... View more

What is the base for the OSS environment? If it is linux-like, is it based on a linux kernel? If so, what version? In case you can't get a UF on the box, the next best option is to gather syslog from your OSS servers using a syslog receiver and UF. If they are linux-based, they should provide some documentation detailing how to set up syslog on them. Gathering syslog using Splunk has been documented in multiple locations (one is provided below). https://www.splunk.com/blog/2016/03/11/using-syslog-ng-with-splunk.html ... View more

Is there a reason, such as storage limitations, that you need to migrate the data off the existing stand-alone instance? The obvious easy path I see is to stand up the new server as a search head, and convert your existing instance into a an indexer. The issue with your current process is that your existing indexed data buckets are not "clustered" buckets, and will not replicate. More info at this link: http://docs.splunk.com/Documentation/Splunk/7.1.0/Indexer/HowSplunkstoresindexes#Bucket_naming_conventions ... View more

This has to do with how MySQL handles user permission. When a user is created in MySQL, it is created with a connecting host to control which hosts a user can connect FROM. Because of this, when connecting to MySQL, the connection driver passes the username AND the host you are connecting from (as 'user'@'host'). In your MySQL database, check your users with permissions. Usually, if you are going to connect from multiple hosts with a user, you will need to specify that when you create the user in the database: mysql> CREATE USER 'user'@'%' IDENTIFIED BY 'password'; In the example, you specify the '%' as a wildcard to denote the user will connect from multiple hosts. Alternatively, you can specify the specific host of dbconnect if your mysql database is more locked down. I would check the 'user' table in the 'system' database on your MySQL instance to make sure your dbconnect user has the proper host assigned to it, or has the host wildcard assigned. ... View more

The TA should ship with props for the [hardware] sourcetype. I checked mine (Splunk 7.0.2 running on MacOS, Splunk_TA_nix 5.2.4), and mine includes the following extracts and evals in the default props.conf: EXTRACT-RealMemory = (?i)MEMORY_REAL\s+(?P<RealMemory>[^ ]*)[ ]? EXTRACT-SwapMemory = (?i)MEMORY_SWAP\s+(?P<SwapMemory>[^ ]*)[ ]? EXTRACT-Unit = (?i)MEMORY_REAL\s+\d+\s+(?P<Unit>\w+)? EVAL-RealMemoryMB = case(match(Unit, "kB"), RealMemory*pow(1024,-1), match(Unit, "KB"), RealMemory*pow(1024,-1), match(Unit, "mB"), RealMemory, match(Unit, "MB"), RealMemory, match(Unit, "gB"), RealMemory*pow(1024,1), match(Unit, "GB"), RealMemory*pow(1024,1), match(Unit, "tB"), RealMemory*pow(1024,2), match(Unit, "TB"), RealMemory*pow(1024,2), match(Unit, "pB"), RealMemory*pow(1024,3), match(Unit, "PB"), RealMemory*pow(1024,3), 1==1, "unknown") EVAL-SwapMemoryMB = case(match(Unit, "kB"), SwapMemory*pow(1024,-1), match(Unit, "KB"), SwapMemory*pow(1024,-1), match(Unit, "mB"), SwapMemory, match(Unit, "MB"), SwapMemory, match(Unit, "gB"), SwapMemory*pow(1024,1), match(Unit, "GB"), SwapMemory*pow(1024,1), match(Unit, "tB"), SwapMemory*pow(1024,2), match(Unit, "TB"), SwapMemory*pow(1024,2), match(Unit, "pB"), SwapMemory*pow(1024,3), match(Unit, "PB"), SwapMemory*pow(1024,3), 1==1, "unknown") EXTRACT-cpu_cores = (?i)CPU_COUNT\s+(?P<cpu_cores>[^ \n]*)? EXTRACT-cpu_type = (?i)CPU_TYPE\s+(?P<cpu_type>[^\n]*)? EVAL-mem = case(match(Unit, "kB"), RealMemory*pow(1024,-1), match(Unit, "KB"), RealMemory*pow(1024,-1), match(Unit, "mB"), RealMemory, match(Unit, "MB"), RealMemory, match(Unit, "gB"), RealMemory*pow(1024,1), match(Unit, "GB"), RealMemory*pow(1024,1), match(Unit, "tB"), RealMemory*pow(1024,2), match(Unit, "TB"), RealMemory*pow(1024,2), match(Unit, "pB"), RealMemory*pow(1024,3), match(Unit, "PB"), RealMemory*pow(1024,3), 1==1, "unknown") When I check the data in splunk, I have the following fields that match up to these props: RealMemory, RealMemoryMB, SwapMemory, SwapMemoryMB, cpu_cores, cpu_type, mem If the add-on is installed on your search heads and indexers, you should get the same field extractions. ... View more

Yes, root will be able to alter indexed data in Splunk by removing (deleting) indexes. This is because root has complete access to the system and can bypass user and group permissions on the server. You should always have proper access controls around root access on your servers. ... View more

I'm assuming you are wanting alerts when someone enters the wrong password trying to log into Splunk. You can create the alert using SPL to search the internal Splunk logs. Here's a basic search, which can be added on as needed: index=_internal component=UiAuth action=login status=failure If you are wanting an alert for a failed logins on a system other than Splunk, then obviously you will need to collect those logs in Splunk and create a search for them. Once you have your search, you can follow the directions in the splunk docs for creating an alert: https://docs.splunk.com/Documentation/Splunk/latest/Alert/Aboutalerts ... View more

We ran into this same issue on RHEL 7. After some digging, we found that daemons loaded by systemd ignore anything in /etc/security/limits.conf AND limits.d/* We wound up extending the service with a file in /etc/systemd/system that sets the limits. Here are the steps we followed: First, create directory /etc/systemd/system/splunk.service.d/ if it does not exist. Then create a file in that directory called "filelimit.conf" The contents of filelimit.conf should look like this (replacing the limit with whatever value you need): [Service] LimitNOFILE=65535 We found this better than trying to modify the init.d script on systems that utilize systemd for startup. ... View more