its benign.. reads before write is known behavior that's why you might see some 404s for bucket uploaded. https://jira.splunk.com/browse/SPL-171553 ... View more

if you want to track over time, collect the timestamp and size fields on some regular basis and output to a lookup or csv file ... View more

Try using the rest api to get the bundle size: | rest splunk_server=local /services/search/distributed/bundle-replication-files | table size ... View more

The eventtype does not exist in any of the windows apps on Splunkbase. If you collect the windows directory services event log you can create a global eventtype that points to the index and source or sourcetype for that data. If you do NOT collect the directory services event log, you can create something like this in any app and share it globally: [wineventlog-ds] search = index=badindex sourcetype=bad Presumably this will not return any data and will run quickly so it should not add much overhead to searches that reference it. Once the search bundle gets deployed to the indexers the errors will go away. ... View more

Look and see if you have a nav.xml in a local directory that might be getting precedence. ... View more

The rest endpoint access is logged via the splunkd_access.log Look for IP addresses that are NOT 127.0.0.1 and those will be the ones where the users hit the REST endpoints from an external server. ... View more

Likely the reason for the poor distribution across indexers is that the SH can read the summarized data very quickly. Unless you have tweaked your ouputs.conf on the SH it will stick to a single indexer for 30 seconds which is probably more than enough time to read the file to EOF. You might try adding something like this to the outputs.conf: [tcpout] forceTimebasedAutoLB=true autoLBFrequency = 10 If the results are really small (couple 1000's) this still might not be small enough interval but I don't think I would go much below 10 seconds. ... View more

Is there a distsearch.conf in the search app that has a blacklist for lookups? ... View more

If you examine the runshellscript.py code there is only one place where it exits with code 2: script = sys.argv[1] if not script: splunk.Intersplunk.generateErrorResults("Empty string is not a valid script name") exit(2) do you also see that message in you logs? I would make sure that the app has been deployed to all members of the SHC. ... View more

This smells like a network connectivity issue. If all indexers are on the same network segment/vlan as the search head the next thing I would look at would be the ulimits for open files if this is a linux environment. All Splunk servers should have a very high value (64k) for this setting. Run the following search: index=_internal source=*splunkd.log ulimit open files if any of your indexers or search heads return a low number (>4k) then you may have found your problem. ... View more

Add a where clause to your select statement that specifies a timestamp field > a week ago. Presumable you could use some sql date function to make the calculation on the fly vs. having to hard code an actual date/time value. ... View more

As mentioned above, btool is your best bet for finding where a setting originates. Be sure to add the debug option so you can find the path: splunk btool inputs list --debug > somefilename.txt ... View more

I looked at the manifest file for the 6.5.1 update (in $SPLUNK_HOME) and it appears that it updates the README file in etc/system/local ... View more

I do not see any milliseconds in the time string from your example. Try dropping the .SSS from the end of the timestamp format. ... View more

Scripted authentication is supported. You should look at the authentication.conf.spec file. There are also samples of scripts located at: $SPLUNK_HOME/share/splunk/authScriptSamples ... View more

etc/system/local over-rides everything I would do this inside an app, search is ok or create your own barebones app and put the settings in a local directory. Some network gear can send to ports other than 514 so you might want to send all devices of one type to a unique port (1514, 2514, etc.) and make it easier on yourself. ... View more

The best practice would be to use a separate syslog or rsyslog server to collect the UPD data and write it to disk. Then use a UF to monitor the files. If that is not possible, then the above technique is what you need to do. There are a couple problems with the transforms.conf settings. The DEST_KEY value for over-riding sourcetype does not begin with an _ Also you will need to add a sourcetype: to the string in the FORMAT statement: FORMAT = sourcetype::cisco:asa From the transforms.conf.spec file: MetaData:Sourcetype : The sourcetype of the event. The value must be prefixed by "sourcetype::" ... View more

Does the forwarder have a useACK=true in the outputs.conf? One of the known side effects of this setting can be data duplication in rare cases. Search for this message in the _internal splunkd.log on the forwarder: WARN TcpOutputProc - Possible duplication of events with ... View more

Leave it on. You probably should always be monitoring the messages and secure logs. ... View more

splunk offline actually stops the indexer. How long do you think the process will take for each indexer? Before you put the cluster in maintenance mode, you might consider increasing the restart timeout value to some number of seconds longer that the process will take: splunk edit cluster-config -restart_timeout 900 Also be sure to take the cluster out of maintenance mode once you are done with the process. ... View more

You need to define what recent means as your time range means everything in splunk. If you need an alert, write the search that finds what you are looking for, then save it as a scheduled search. You must then specify the time period you consider recent and how often you want to run the search. You might consider running the search 1x per day (8 am or whenever) and then look back over the last 24 hours (-24h) until now. ... View more

What do you mean by servers? The splunk indexers that are returning the data or the actual host names returned by the metadata? If you want to filter by hosts, I like to use a regex statement since often host names have some pattern that can be used (i.e. wxxx for windows, xxxpxxx - production, etc.) This is easy with regex: | regex host!="^w" or | regex host!="np|dev" Just remember that regex is case sensitive, so you may need to use "(?i)np|dev" to make it case insensitive. ... View more

You mentioned that you are using a heavy forwarder on the syslog server. I took a look at the app and in the props.conf section there is a set of transforms that operates on the syslog sourcetype that changes the sourcetype for the different types of events that the F5 generates: TRANSFORMS-sourcetype=f5-dcfw,f5-syslog,f5-access This means that you will have to install the app on the heavy forwarder so it can do this operation before the data gets indexed. I didn't see any index specified in the app, so not sure what index you are putting the data into, but if you don't have rights to search all indexes by default, you might consider modifying the macros.conf file (on the search head) and putting index=foo (the index where the f5 data is) in front of the sourcetype. Something like: definition = index=foo sourcetype="F5:AFM:Syslog" After that as long as you have the app installed on your search head, you should be good to go. , ... View more