About twinspop

twinspop · ‎07-14-2016

That's (at least partly) because crcSalt uses the path of the file to determine the hash that represents the file. Every time you move it to a new directory, splunk says "hmm, the hash has changed! new file! eat it up!" Normally, splunk only considers the first 256 bytes of a file to ID it. If that is the same, it will check mod time, seek to the last read spot, and input from there.

twinspop · ‎07-14-2016

crcSalt is causing your problem. You do not want to use it in a case like this.

twinspop · ‎07-14-2016

You should share your inputs.conf stanza for these files

twinspop · ‎07-13-2016

Correct. The tstats command will follow your time restraint. This command will initially find all hosts that have logged any data in the last 3 days in any index. The filtering will then only show those that stopped more 86400 seconds ago.

twinspop · ‎07-13-2016

I would probably use tstats instead of metadata , and just set the time selector to last 3 days: | tstats latest(_time) as lastTime by host | eval age=now()-lastTime | search age > 86400 | sort age d | convert ctime(lastTime) | fields age,host,lastTime Metadata can yield unexpected results when you set a timeframe. EDIT: changed to days from hours. Doh!

twinspop · ‎07-13-2016

The Splunk-provided TA for *NIX app supports several flavors of UNIX, including Mac OS X.

twinspop · ‎07-12-2016

some search [search some other search earliest=-15m] m = minutes h = hours d = days w = weeks mon = months

twinspop · ‎07-11-2016

Okay. Wow. As Jobe would say, "I've made a huge mistake." I assume the same holds true for user files and dirs as well? When I transferred this app from the SHP to SHC, I tarred up all the $splunk/etc/users/theAppInQuestion directories and untarred them on the Deployer in $splunk/etc/shcluster/users. Apparently this was a bad idea. The way I see it now, I should have untarred on each SH. Ug, got some cleanup to do.

twinspop · ‎07-11-2016

Is it up to us to find 3rd party utils to synch the files pushed directly to SHs? For example, I want to add a new stanza in transforms.conf. As far as Splunk is concerned, I need to copy that file individually to each SH and restart? Or is there a better way?

twinspop · ‎07-11-2016

Our splunk install has a (new) Search Head Cluster. Previously we were running Search Head Pooling. I'm struggling to figure out how to manage the content of users' Splunk apps. With SHP, the users had access to delete and change objects from any search head in the pool. Now, the SHC promotes artifacts from local to default, and users cannot, for example, remove an unwanted dashboard. Or update a lookup file. I've tried to force the requested changes on the Deployer, but most often the changes are not pushed out to the Search Heads in the SHC. For a specific example, let's use the lookup file situation. On the SHP, to update the file, they would upload a new one, delete the old one, and change the perms on the new one to be "app level," promoting it into place. How can they accomplish this task in SHC? Another example: User wanted a view removed. Since they can't do this, I went to the Deployer, deleted the XML file, and issued a splunk apply shcluster-bundle command. However, the file was not removed from SHC members. What is the proper way to do this?

twinspop · ‎07-11-2016

| eval newfield=if(field1=0 and field2=0,1,null)

twinspop · ‎07-06-2016

Apparently this bug introduced in 6.3 is still active: https://answers.splunk.com/answers/329518/why-do-scheduled-searches-randomly-stop-running-in.html I used the workaround listed there (in limits.conf) and the monotonically increasing usage stat for the user has returned to 0.

twinspop · ‎07-01-2016

4 node SHC running 6.4.1, Linux 64-bit. Very lightly used (still working on migrating from the old system). 1 user has 3 searches that fire at midnight, and 3 more that fire hourly. The searches run quickly, under 1 second in every case I've tested. And we've put in a window for all of them: 5 minutes for the hourly, and 3 hours for the nightly. All throughout the day I see messages from SHCMaster: 07-01-2016 10:00:26.234 -0400 ERROR SHCMaster - Search not executed: Your maximum number of concurrent searches has been reached. usage=12 quota=12 user=jsmith. for search: nobody;docker_search;Agency Notice Worker Notification Error They were logging 100s of times per hour. And reporting usage=53, which was insane. No, we didn't have 50 searches running. I performed a rolling restart of the cluster, and it settled down for a few hours. But then they came back. Not as frequent, but still there. Most concerning of all, it is causing missed runs of the scheduled searches. The DMC Search Activity Page and Scheduler Activity page show between 0 and 3 active searches at the same time SHCMaster is reporting this problem.

twinspop · ‎06-27-2016

Confirmed, yes. The box is checked. A sample search, very simple: error | stats count by host but it hasn't happened since June 10th, a day we pruned lots of the per minute searches we had. Could it be related to overrun quotas? If so, strange message.

twinspop · ‎06-08-2016

06-08-2016 21:11:02.773 -0400 ERROR SavedSplunker - savedsearch_id="UserX;search;ss_index_delay_times", message="Error in 'summaryindex' command: Permission denied to index 'summary'.". No actions executed The search is owned by UserX, and scheduled to run as 'owner'. UserX has plenty other searches that are running and saving to this same summary index. This very search runs perfectly fine most of the time. UserX is admin level. It's running on a solo search head talking to 3 indexers. Index summary is defined on the SH, but the SH is set to forward to the indexers. No local indexing. Linux, running 64-bit 6.4.1.

twinspop · ‎05-10-2016

index=linux sourcetype=restartsplunk splunk is running | eval status=if(match(_raw,"is running"),"up","down") | table _time host status | sort _time | streamstats global=false current=false window=1 last(status) as previous_status by host | where status!=previous_status This search will create a table with the times of all up and down records. After a sort to get it in ascending time order, it adds a column that shows the previous seen state for that host. Finally, we only show those times when state changed, either up to down or down to up. You could change the where clause to status="up" and previous_status="down" to see only the transitions from down to up.

twinspop · ‎05-06-2016

Not pretty, but it's doing what I need: Provide hostname, IP, Last Phone Home time, phone home interval and (bonus) last time data was seen, if you have access to the indexes where data was written. It requires that you run it locally on the DS, or have the DS listed a search peer on the server you are searching on. | rest splunk_server=deploymentserver /services/deployment/server/clients | fields hostname ip *Phone* | eval "Last Phone Home, Minutes Ago"=round((now()-lastPhoneHomeTime)/60,2) | fields - lastPhoneHomeTime | eval hostname=lower(hostname) | join type=outer hostname [ | metadata type=hosts index=*| eval "Last Data, Hours Ago"=round((now()-lastTime)/60/60,2) | table host, "Last Data, Hours Ago" | rex field=host "(?<hostname>[^\.]+)" | fields - host | eval hostname=lower(hostname) ] | where isnotnull(ip) | fillnull value="no data, or index not available"

twinspop · ‎05-06-2016

Thanks for the hackerific solution. That might work for me (assuming it works at all :-). I've been toying with the REST endpoint. With 3000 clients it returns a metric ton of information. Will need to figure out how to present something similar to the DSM page from there.

twinspop · ‎05-06-2016

We have many group-admins that need to be able to check on their deployment clients, but I don't want to give them write access to the DS. I could give them all access to _internal and parse out phoneHome connections from there, trying my best to duplicate the DS interface. But is there a better way? I tried limiting their capabilities in a new role on the DS, but without admin, the Forwarder Management app isn't accessible. Thanks, Jon

twinspop · ‎05-05-2016

Curious about your use of dedup and table vs stats . With this snippet replacing that section of your search: | stats latest(version) as version latest(arch) as arch latest(os) as os latest(build) as build by hostname | rex field=hostname "(?<hostname>[^\.]+)" ... the search takes about 15% less time for me. Not a huge savings, for sure, but still. Any advantage to using depup + table ?

twinspop · ‎05-02-2016

Yep. Event size was important to my system at one point so I set-up an accelerated data model using the same eval you have shown above. With the ADM it's easy to grab stats based on sourcetype, source, index and/or host. Once the need passed, I disabled the acceleration.

twinspop · ‎04-22-2016

MAX_TIMESTAMP_LOOKAHEAD is telling splunk to only look 30 bytes into the message. Your timestamp is well beyond the 30 byte marker.

twinspop · ‎04-21-2016

For the overall license usage and total available, explore the REST API. I used that as the basis for a fill-gauge panel using the below search. The "Danger Zone(tm)" adjusts based on time of day. | rest splunk_server=local /services/licenser/pools/your_pool | fields title effective_quota used_bytes | eval used=round(used_bytes/(1024*1024*1024),2) | eval h=tonumber(strftime(now(),"%H"))/24 | eval danger=round(h*effective_quota/(1024*1024*1024),0) | eval max=round(effective_quota/(1024*1024*1024),0) | eval base=0 | eval gauge_top=max+(max*.01) | gauge used base danger max gauge_top

twinspop · ‎04-20-2016

Try this: [your_sourcetype] TIME_PREFIX = "start":" TIME_FORMAT = %s.%3N

twinspop · ‎04-20-2016

Unfortunately, host_regex only applies to the source path not arbitrary log contents.

Posts	609
Solutions	72
Karma Given	384
Karma Received	232
Member Since	‎06-09-2010

Online Status	Offline
Date Last Visited	‎04-07-2021 02:44 PM

Kafka add-on: Unable to initialize modular input

With 8.1.2 or later should we still use an HEC tie...

Which users use the REST API?

Can I set-up Splunk to replace a syslog server?

How does monitor:// handle windows shortcuts to di...

What is the proper way to make user-prefs settings...

How do I block GUI messages about missing indexes?

It is possible to "tag" all data coming into a par...

How to redirect some data coming into an indexer (...

How to disable schedules for all searches for a pa...

Re: How to index only one day of data from a batch...

Re: How to index only one day of data from a batch...

Re: How to index only one day of data from a batch...

Re: Search to Identify when a host stops sending l...

Re: Search to Identify when a host stops sending l...

Re: Is there any way to monitor CPU on Mac OS?

Re: Time modifiers in a subsearch

Re: How do you manage the content of users' Splunk...

Re: How do you manage the content of users' Splunk...

How do you manage the content of users' Splunk app...

Re: Can I use an eval if statement to create a new...

Re: What would cause SHCMaster to log "Search not ...

What would cause SHCMaster to log "Search not exec...

Re: What causes this scheduler failure? "Error in ...

What causes this scheduler failure? "Error in 'sum...

Re: How to search if a host is down and when it ca...

Re: How do I give read-only access to the deployme...

Re: How do I give read-only access to the deployme...

How do I give read-only access to the deployment s...

Re: Query to return all forwarder versions and the...

Re: How to get average event size...

Re: Why does splunk think it can't parse my timest...

Re: How to create a timechart on license usage to ...

Re: How to configure props.conf for a Unix timesta...

Re: How to convert computer name to host name?

Join the Conversation