This isn't really a Splunk question, more a particular network situation.
Refer to https://www.akadia.com/services/ssh_putty.html for details on how to configure PuTTY to establish a tunnel between your current machine and the bastion, and the bastion to the cluster master. This tunnel will exist as a local port on your machine which forwards connections (via the bastion) to the cluster master's port 8000.
SO you should end up with something like
L8000 clustermasterip:8000
... View more
If you have a timestamp you can work with
| eval month=strftime(_time,"%b")
Otherwise, you could tidy up the if statement by using a case statement instead
| eval month=case(
MM="01","Jan",
MM="02","Feb",
MM="03","Mar",
MM="04","Apr",
MM="05","May",
MM="06","Jun",
MM="07","Jul",
MM="08","Aug",
MM="09","Sep",
MM="10","Oct",
MM="11","Nov",
MM="12","Dec",
1=1, "INV")
... View more
You are using strftime , which is converting your times into text strings, which you can't do maths operations on.
What format is time_of_last_change ? Is it a unix timestamp? If so, you can calculate the difference by doing eval diff=relative_time(now(), "-1h@s") - time_of_last_change .
Otherwise, you most probably want to use the strptime command, which converts times in text format into a unix timestamp, which is seconds. From there, you can calculate the time difference.
... View more
In your query, you are not correlating your CSV data with the original indexed data. You may need to do a stats or a join to combine the indexed data with your CSV data.
Alternatively, have you tried using your CSV as a lookup?
index=s1 | lookup cosco_mapping.csv city AS office OUTPUT lattitude longitude
... View more
When you created your custom command, did you specify that it should be a streaming command?
http://dev.splunk.com/view/python-sdk/SP-CAAAEU2#streamingcommand
http://docs.splunk.com/DocumentationStatic/PythonSDK/1.6.5/searchcommands.html#splunklib.searchcommands.StreamingCommand
... View more
All your questions are covered in the docs, http://docs.splunk.com/Documentation/HadoopConnect/1.2.5/DeployHadoopConnect/ExporttoHDFS
Compression Level
Use the slider bar to determine file compression. 0 means no compression and 9 give you the maximum possible compression. The higher the compression level is set, the more slowly files are written. The higher the compression, the smaller the size of the exported files in hdfs will be. Also note that higher compression may also mean slower retrieval of data
How does partitioning work?
Partitioning is a process by which the export data is placed in a dynamic directory structure based on the values of certain event fields. You can choose how exported data is partitioned. It can be partitioned by any of the fields present in the event.
Hadoop Connect exposes the following out-of-the-box partitioning variables:
Date
Hour
Host
Source
Sourcetype
When you are creating an export job, you can select one or more of these partition variables in the user interface.
Splunk will create directories based on these partions in hdfs. So if you select Date, Hour and Sourcetype, Splunk will create the following directory structure for your data /2018/11/06/00/WinEventLog_Security/ to store the various files containing your exported data
Creating your own partitions
In addition to the out-of-the-box partitioning variables, you can use any field to compute a partitioning path into the events (results) to be exported. Use the special field _dstpath . For example, to export your search results into the path <base-path>/<date>/<hour>/<app> , use the following search string:
search ….. | eval _dstpath=strftime(_time, “%Y%m%d/%H”) + “/” + app_name
You can perform other types of preprocessing of data in the Splunk platform (lookups, field extractions, evaluate other fields, and so on) or choose to export it in raw format
Schedule Export
The Schedule Export is the only way to export indexed data into hdfs.
... View more
I suspect your time parsing is incorrect.
Do you have the correct timezones for your data?
Are they log sources, forwarders, indexers and search heads all running NTP and are time syncing correctly?
Does the search work for one user, and if so, what is the timezone that they are using?
When making a scheduled search, it is often good to have some latency in the data you are looking at. Try putting earliest=-5m@m latest=-35m@m to make sure any events that are delayed have 5 minutes to be ingested before they are searched.
... View more
Can you post a better example of your data, your sample tells us nothing about the data and what you are attempting to replace. If we could see more, we could suggest a better SED expression for you to use.
... View more
Oracle Linux is essentially a repackaged RedHat Linux, so the RPM versions of Splunk Enterprise and the Universal Forwarder should be compatible. You can also use the tgz version if you prefer.
... View more
If you want a dashboard with many panels, you will need to expose multiple reports and create a HTML page that embeds all of them.
Converting a dashboard to HTML will not help. You are exposing reports, not dashboards.
... View more
You can use Report embeding to be able to show a Splunk report on a separate server, see http://docs.splunk.com/Documentation/Splunk/7.2.0/Report/Embedscheduledreports
... View more
source="splunk_test1.txt" host="Sriharshas-MacBook-Pro.local" sourcetype="temp" | table x, y | sort y | stats max(x) by y
Instead of max(x) , you could also try sum(x) , depending on what that data represents
... View more
The issue you are having is because you have a | search Thread:1 , all other threads are being discarded after that point.
You probably need to incorporate the Thread:1 etc... into your rex statement. Can you provide some log samples.
... View more
Have you confirmed that you can run Map Reduce jobs from the search head via the command-line, i.e., not using Splunk.
Confirm that you have Hadoop properly configured. My customer was missing the yarn-site.xml and core-site.xml from the /opt/hadoop/etc/hadoop/ directories, and their deployment exhibited the same issue as the one you have.
http://docs.splunk.com/Documentation/HadoopConnect/1.2.5/DeployHadoopConnect/Setupcompressedfiletypes
Also refer to the following question on stack overflow, https://stackoverflow.com/questions/43425678/application-failed-2-times-due-to-am-container-exited-with-exitcode-1 . Again, check that hadoop actually works from the CLI, check all CLASSPATH and PATH variables are correct, using the splunk user.
... View more
To enable SSL for your web interfaces, you need to have the following in web.conf (http://docs.splunk.com/Documentation/Splunk/latest/admin/Webconf)
[settings]
enableSplunkWebSSL = true
You will need to restart for this to take effect.
You can also do this via the GUI, as described at http://docs.splunk.com/Documentation/Splunk/7.2.0/Security/TurnonbasicencryptionwithSplunkWeb
... View more
You will need to use prestats mode with tstats, see https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Tstats
| tstats prestats=true summariesonly=true count from datamodel="Network_Traffic.All_Traffic" where All_Traffic.dest_port=22 by All_Traffic.dvc, All_Traffic.action
| rename All_Traffic.dvc as Device, All_Traffic.action AS Action
| stats sum(count) by Device, Action
... View more
Have you indexed these Excel files in Splunk already? You may need to convert them to CSVs, or use some other approach ( https://simonduff.net/processing_excel_xlsx_files_with_python/ )
Once you have them in Splunk, its very simple to search for them
index=* bum_id a62f1ede-e3c2-418a-8376-945d94288c56
or
index=* source="*/file1.csv" OR source="*/file2.csv" OR source="*/file3.csv" bum_id a62f1ede-e3c2-418a-8376-945d94288c56
... View more
You shouldn't need to clear the fishbucket when you update inputs.conf.
When you clear the fishbucket, you delete everything that Splunk 'remembers' about what files it has already read. So when you next start Splunk, it re-reads all the files again.
Otherwise, are your files being rolled in that directory? crcSalt=<SOURCE> means that it treats each filename as a unique, regardless of the contents. So if you have a file called log/file.log which gets rolled to log/file.log.1 , it will get re-read. I don't think this is applicable here, but just something to double check.
... View more
Check the following TA for Linux auditd, https://splunkbase.splunk.com/app/4232/
There is a companion app, https://splunkbase.splunk.com/app/2642/, that will make use of the collected data for some pretty dashboards.
... View more
When you first ran splunk, you ran it as the root user. So Splunk created a number of files owened by root .
You need to run chown -R splunk:splunk /opt/splunkforwarder
... View more
Probably the best resource for understanding ExtremeSearch is George Starcher's Blog series, starting at http://www.georgestarcher.com/tag/extreme-search/
To me, it appears as though the context generation scripts are not running, or if they are running, aren't producing the right values.
Check to ensure Network - Port Activity By Destination Port - Context Gen is enabled and running regularly.
That context gen runs the following to generate results
| tstats `summariesonly` count as dest_port_traffic_count from datamodel=Network_Traffic.All_Traffic by All_Traffic.dest_port,_time span=1d | `drop_dm_object_name("All_Traffic")` | `context_stats(dest_port_traffic_count, dest_port)` | search size>0
Run this for -30d@d and check it is producing results.
... View more
As an alternative to changing the index on the recipient HF/Indexer, you may try using sourcetype cloning. The caveat will be that the sourcetype will end up different on each cluster (although you could put additional config there to change it back).
On the HF,
props.conf
[original_sourcetype]
TRANSFORMS-clone = clone_sourcetype
[sourcetype2]
TRANSFORMS-change_index = change_index
transforms.conf
[clone_sourcetype]
CLONE_SOURCETYPE = sourcetype2
REGEX = .
[change_index]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = web
... View more
Suggest you create a new question, but yes, just add something like the following to that inputs.conf file and restart the splunk service
[monitor:///path/to/new/directory/or/file.txt]
disabled = false
index = default
... View more