Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About Log_wrangler
Log_wrangler
Builder
Member since:
11-09-2017
06-05-2020
Community Statistics
| Posts | 265 |
| Solutions | 9 |
| Karma Given | 0 |
| Karma Received | 18 |
| Member Since | 11-09-2017 |
Activity Feed
- Got Karma for How to Eval urldecode and then regex?. 08-03-2021 02:51 AM
- Got Karma for base64 decoding fails because of padding. 06-25-2021 03:05 AM
- Got Karma for Getting an error with Service NOW add-on.... 08-18-2020 06:06 AM
- Got Karma for Where is the installation and configuration documentation for the Elasticsearch Data Integrator app?. 06-05-2020 12:50 AM
- Got Karma for Re: Can you help me with my query involving a saved search and summary indexing?. 06-05-2020 12:50 AM
- Got Karma for Can you configure a Universal Forwarder output to send to two separate Heavy Forwarders?. 06-05-2020 12:50 AM
- Got Karma for How to use the current Deployment Server to configure remote UFs with a new Deployment Server IP?. 06-05-2020 12:50 AM
- Got Karma for Where is the installation and configuration documentation for the Elasticsearch Data Integrator app?. 06-05-2020 12:50 AM
- Got Karma for Re: How do I upgrade a non-clustered search head to 7.1?. 06-05-2020 12:49 AM
- Got Karma for Splunk Add-on for Microsoft Cloud Services, splunk is not able to read .gz blob files, how do I fix this?. 06-05-2020 12:49 AM
- Got Karma for Re: how to calculate a running average of events by user for 'All time' (Part 2). 06-05-2020 12:49 AM
- Got Karma for how to calculate a running average of events by user for 'All time' (Part 2). 06-05-2020 12:49 AM
- Got Karma for Can splunk monitor a google drive folder as a data input?. 06-05-2020 12:49 AM
- Got Karma for Create a sourcetype to index map. 06-05-2020 12:49 AM
- Got Karma for Re: How to send syslog data to the indexer and another TCP listener?. 06-05-2020 12:49 AM
- Got Karma for Getting an error with Service NOW add-on.... 06-05-2020 12:49 AM
- Got Karma for Getting an error with Service NOW add-on.... 06-05-2020 12:49 AM
- Got Karma for Invalid key in stanza. 06-05-2020 12:49 AM
- Posted Re: How do you set the License Usage App to use UTC time ? on All Apps and Add-ons. 06-06-2019 03:01 PM
- Posted How do you set the License Usage App to use UTC time ? on All Apps and Add-ons. 06-05-2019 05:48 PM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 | |||
| 2 | |||
| 0 | |||
| 0 | |||
| 1 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 |
02-14-2018
07:31 AM
Specify a target group for a hostname which consists of a single receiver.
[tcpout:group2]
server=myhost.Splunk.com:9997
am I correct from this example, it is possible?? just need confirmation and advice if there is any kind of problem that can occur with using a DNS name. But in my case, its going to a 3rd party (nifi processor) so the name and port will be differente.
thank you.
... View more
02-14-2018
07:24 AM
I have a scenario where data from a fwdr needs to go to a dns name (load balancer) instead of IP.
Please advise if this is possible (with an example conf please).
Thank you
... View more
02-13-2018
11:01 AM
I have a few nifi nodes (on EC2 instances) running in AWS.
I use nifi to process data into s3.
I am looking for a way to splunk the /nifi-app.log (logs) for warning, errors, failures, etc.
I am thinking about putting a UF on each of the nifi nodes/EC2 and sending the data to the indexers.
Any advice on this scenario is appreciated.
Thank you
... View more
01-30-2018
09:49 AM
I forgot to enable all of the data inputs under:
Data inputs » Enable ServiceNow database table inputs.
After enabling, I am getting events in main as expected.
Please convert to an answer so that I can accept.
Thank you
... View more
01-30-2018
08:03 AM
So you are correct, I have events under
index=_internal ta-snow OR ta_snow OR servicenow OR snow
but nothing under 'main'...
any other ideas?
Thank you
... View more
01-30-2018
06:56 AM
Thank you for the advice.
I am actually testing the SNOW-TA on a heavy Fwdr to send the SNOW data to the indexers > main. And to make things more difficult I don't have SSH access to the box, only GUI to the Fwdr.
I will take a look at the search above and let you know.
Thank you
... View more
01-29-2018
09:02 AM
3 Karma
I configured the SNOW add-on per documentation.
Under manage app >Splunk Add-on for ServiceNow>setup I entered https://url_to_serviceNOW.com.
An api account was created in ServiceNOW (usr/pwd) and that was entered as well, under (manage app >Splunk Add-on for ServiceNow>setup).
I have read a number of posts already related to the error I am getting :
Encountered the following error while trying to update: Error while posting to url=/servicesNS/nobody/Splunk_TA_snow/service_now_setup/snow_account/snow_account
However, I was not understanding standing what the cause and resolution is.
Anyone able to explain if this is a Splunk issue or Service NOW issue? I followed the documentation on the SN add-on.
Thank you
... View more
01-23-2018
11:25 AM
Thank you for confirming.
So in general the file structure/syntax will remain the same rolling from warm to cold, and I should see the same file naming (e.g. db_1234567890_0987654321_1234) unless it is "replicated" then rb_ will precede the files...
Please correct me if I am misunderstanding.
Thank you
... View more
01-23-2018
10:19 AM
I would presume that hot/warm db data files and colddb data files would be the same but cold would be older?
... View more
01-23-2018
09:09 AM
I am viewing my db and colddb locations.
looking in my ... /db I see the usual db_1234567890_0987654321_1234 files
but when I look in my .../colddb I see only rb_1234567890_0987654321_123_... - ...- ...-tmp.
The data I am using is from either a cluster or pool of indexers so I am assuming the rb is replication bucket?
What does data from db look like when it rolls to colddb (in general, just trying to understand the structure/syntax of normal)?
Will dbinspect show cold state info too?
Thank you
... View more
- Tags:
- colddb
01-12-2018
02:30 PM
FYI, I was able to read a test file.txt from the /s3fs dir, but as a "data Input'
I could read the file.txt via data inputs > files & directories > new (then select the /s3fs/file.txt)
Of course this would need to be automated to input loads of files.... have not worked that out but any suggestions appreciated.
... View more
01-12-2018
10:48 AM
Your suggestion is probably the best solution at this point.
My current scenario was a test to see if it would read, and apparently it will not (as you have mentioned the s3fs is slow, also object based, and not listed as supported).
For those interested I started another thread (title of question below) to see if Splunk 7.0 remotePath may be a solution.
"has anyone successful setup the remotePath option in indexes.conf in Splunk 7.0 to work with indexed data in s3?"
... View more
01-12-2018
10:38 AM
I was trying to search copies of indexed data in S3.
Has anyone had luck with this scenario using remotePath??? I know it says not supported but is it functional at this point?
indexes.conf for splunk 7.0
[https://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/Indexesconf]
remotePath =
* Currently not supported. This setting is related to a feature that is
still under development.
* Optional.
* Presence of this parameter means that this index uses remote storage, instead
of the local file system, as the main repository for bucket storage. The
index processor works with a cache manager to fetch buckets locally, as
necessary, for searching and to evict them from local storage as space fills
up and they are no longer needed for searching.
* This setting must be defined in terms of a storageType=remote volume
definition. See the volume section below.
* The path portion that follows the volume reference is relative to the path
specified for the volume. For example, if the path for a volume "v1" is
"s3://bucket/path" and "remotePath" is "volume:v1/idx1", then the fully
qualified path will be "s3://bucket/path/idx1". The rules for resolving the
relative path with the absolute path specified in the volume can vary
depending on the underlying storage type.
* If "remotePath" is specified, the "coldPath" and "thawedPath" attributes are
ignored. However, they still must be specified.
Any advise or lessons learned is appreciated.
Thank you
... View more
01-12-2018
10:27 AM
it was a copy of warm and cold.
... View more
01-12-2018
08:45 AM
sorry sec-policy does not permit to post actual data thx
... View more
01-12-2018
08:41 AM
After reviewing, some other posts...
It is quite possible that the s3 object based data is just not compatible (with Splunk) without some custom code making it readable for splunk.
I am using an old version of Splunk (i.e. 5.x).
I am thinking that I will try Splunk 7.x and see if it can read indexed data from a remote s3 location.
Please advise if you have any more insight on this. If/when I get results, I plan to share lessons learned.
Thank you
... View more
01-12-2018
08:35 AM
This is a standalone splunk instance on RHEL7 on the EC2 AWS instance.
I created a custom index which points to the s3 path...
When I restarted after creating the indexes.conf file for this index, I got this error
error message for /my_s3fs_mount_directory/...
homePath '/my_s3fs_mount_directory/index_name/db' is in a filesystem that Splunk cannot use. (index=index_name)
Checking indexes...
homePath '//my_s3fs_mount_directory/index_name/db' is in a filesystem that Splunk cannot use. (index=index_name)
Validating databases (splunkd validatedb) failed with code '1'.
... View more
01-08-2018
02:50 PM
I have previously indexed data uploaded to an s3 bucket.
I installed Splunk (full version) on an EC2 (RHEL7).
I (persistently) mounted the s3 bucket to the EC2 instance (with FUSE).
I can see all the data when I change to my_s3fs_mount_directory,
(e.g. /my_s3fs_mount_directory/index_name/db_1234567_123456_1234/rawdata/journal.gz)
My question is how I should edit the indexes.conf correctly, so that my new indexer sees this data and doesn't accidentally overwrite the existing data in my path by accident.
Here is what I have so far (in /opt/splunk/etc/system/local/)
[myindex]
homePath = /my_s3fs_mount_directory/index_name/db
coldPath = /my_s3fs_mount_directory/index_name/colddb
thawedPath =/my_s3fs_mount_directory/index_name/thaweddb
maxDataSize = 10000
maxHotBuckets = 10
The index is visible but no data in results.
Is there anything else I need to do or another conf I would also need to edit?
Any advice is appreciated.
Thank you
... View more
- Tags:
- aws
- indexes.conf
- s3
01-02-2018
02:09 PM
Thank you for the reply.
I should mention that the data in the S3 bucket is previously indexed data (uploaded via snowball). So I wanted to seamlessly access that data with the aws/ec2/rhel7/splunk enterprise deployment as before. This splunk instance will act as an indexer.
Do you see any issues with this scenario?
I will dig into your links.
Thank you.
... View more
01-02-2018
12:56 PM
Hi All,
I know this question leans more to AWS support but I thought I would poll the community for any insight.
I have an EC2 (RHEL7 OS) rolling in AWS and I am in the midst of installing Splunk Enterprise.
I have some Splunk data saved in an S3 bucket, but I am apprehensive about mounting the S3 data to the EC2 instance. I am worried that if not done properly Splunk might overwrite the data in S3.
Does anyone have experience with this?
Are there any caveats regarding the way S3 needs to be connected to Splunk's OS?
I have seen two ways (so far) "IAM role to access S3 buckets" and "Fuse" to connect the data to the OS.
Has anyone successfully tried either way with Splunk?
Thank you
... View more
12-29-2017
10:57 AM
I came up with this
index=* |stats values(sourcetype) dc(sourcetype) by index
Please let me know if there is a better way...
to clarify I want to see a count unique sourcetypes per index, for example main has sourcetype A B C D E, so the count would be 5.
... View more
12-29-2017
10:43 AM
Thank you for the reply. I will check the props.conf when I am granted access.
I can get by with
index=* |stats values(sourcetype) by index
but I forgot how to write the code to count the number sourcetypes per index, (not the events per sourcetype)... any advice is greatly appreciated. I have multiple sourcetypes per index and at least 24 so far.
Thank you
Thank you
... View more
12-29-2017
10:07 AM
1 Karma
I am exploring an unfamiliar Splunk Enterprise deployment.
Normally I use:
|tstats values(sourcetype) WHERE index=* by index
to get a quick reference map of sourcetype to index, but it only shows a few indexes.... (4 total)
When I run;
index=* |stats values(sourcetype) by index
I get more (24 total).
When I run:
|metadata type=sourcetypes index=*
I get sourcetypes but not index values in a column
Please advise what I might use to get a comprehensive list of indexes and sourcetypes.
Any explanation for the different results is much appreciated too.
Thank you
... View more
12-28-2017
12:09 PM
Thank you, that is what I was concerned about....
... View more
12-27-2017
10:42 AM
Thank you for the clarification.
What happens if the following is in place?
[tcpout]
defaultGroup=nothing
... View more
- « Previous
- Next »
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
06-05-2020
02:03 AM
|
