Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
×
Are you a member of the Splunk Community?
Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
- About grijhwani
grijhwani
Motivator
Member since:
04-25-2013
06-05-2020
Community Statistics
| Posts | 413 |
| Solutions | 52 |
| Karma Given | 52 |
| Karma Received | 178 |
| Member Since | 04-25-2013 |
Activity Feed
- Got Karma for Re: How can i change the time format in Splunk web?. 02-10-2021 09:29 AM
- Got Karma for Re: How can i change the time format in Splunk web?. 11-24-2020 09:36 AM
- Karma Re: How to install multiple instances of Splunk 6.4 on a single Linux server? for jkat54. 06-05-2020 12:48 AM
- Karma Re: Distributed Indexer Hardware for Yasaswy. 06-05-2020 12:47 AM
- Karma Re: Why is syslog right into Splunk so bad/wrong? for alacercogitatus. 06-05-2020 12:47 AM
- Karma Green background for accepted answers for vqd361. 06-05-2020 12:47 AM
- Karma Re: How do I search a csv file created via Outputlookup? for mshapirovp. 06-05-2020 12:47 AM
- Karma Re: Duplicate events because of hosts writing to the same log file shared on a NFS for echalex. 06-05-2020 12:47 AM
- Karma Re: Free licence behaviour. for piebob. 06-05-2020 12:47 AM
- Karma Re: Using Sudo as part of an input command for MuS. 06-05-2020 12:47 AM
- Karma Re: How to make Splunk read my files faster? for MuS. 06-05-2020 12:47 AM
- Karma Re: Hostname from GUID for lguinn2. 06-05-2020 12:47 AM
- Karma Re: What are suggested approaches/steps for Syslog to switch to another heavy forwarder when primary goes down? for MuS. 06-05-2020 12:47 AM
- Karma Re: What are suggested approaches/steps for Syslog to switch to another heavy forwarder when primary goes down? for musskopf. 06-05-2020 12:47 AM
- Karma Is it possible to have a search string so long that it can't be parsed in Splunk 6.0 or higher? for rroberts. 06-05-2020 12:47 AM
- Karma Re: What are the steps to migrate a Splunk instance from one Unix machine to another? for ppablo. 06-05-2020 12:47 AM
- Karma Re: How splunk assigns timestamp for richgalloway. 06-05-2020 12:47 AM
- Karma Re: Who owns and provides support for Splunk apps available in the market? for piebob. 06-05-2020 12:47 AM
- Karma Search filter shortcut for "NOT" for paterler. 06-05-2020 12:47 AM
- Karma Is there a patch planned for OpenSSL Security Advisory [6 Aug 2014]? for khyoung7410. 06-05-2020 12:47 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 1 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 3 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 2 | |||
| 7 |
09-19-2016
10:55 AM
I downvoted this post because that is not a solution. It requires that the pid file be removable. The process you link to will fail because the file-system has been set to read-only by the kernel, a situation which implies a significant system fault which needs to be addressed. The very problem being explicitly reported by the start-up is that the pid file cannot be removed.
... View more
09-19-2016
09:07 AM
This is a 3 year-old question. You should, really, if you want to garner a new answer, open a new question.
That is largely irrelevant, though. As it is all of my previous responses stand. There's nothing new or different about your problem that isn't already addressed above. The file-system in read-only mode, when to have written the PID file it must have previously have been read-write suggests a file-system error was detected by the kernel. You need to check your logs and dmesg buffer for disk and file-system errors and remediate the problem. Until you do, the system is highly suspect and at risk. You may want to consider backing it up before you do, but you're going to need to reboot (with the understanding that if this is the first manifestation of a bigger fault, it may not come back). Read my answers above. This isn't a Splunk problem, it's a fundamental sysadmin problem.
... View more
06-03-2016
05:48 PM
2 Karma
Years on and this is - as far as I can tell - haunting us.
The choice of 24 hour clock isn't regional. Nor is, say, the preference for ISO dates. There needs to be an easy way to set the base standard for the local search instance.
... View more
06-03-2016
05:43 PM
You should still be able to perform searches on Splunk's own internal data sources, even if general indexes/sources are not available.
Try this to look at where your data sources are:
index=_internal (source=*/metrics.log* OR source=*\\metrics.log*) group=per_sourcetype_thruput
| timechart span=1d sum(kb) by series
| addcoltotals
| addtotals
The resulting table should show you on a daily basis what is causing your excess.
Also you need to be aware that access to the general data will remain blocked as long as 3 or more days of excess remain in a window of the last 30 days. With the Enterprise licence this 5 excess days, which could explain why things broke when you switched to the Free licence - if the licence was expired by 4 days you would have exceeded a cap of zero by 4 days, but the interface would still work. On switching to free you only have a leeway of 2 over-capacity days in 30, which would cause you to be excluded from general data. The only solution to that is to address the excessive data, and then wait until the window is no longer exceeded - worst case scenario, 28 days.
... View more
06-03-2016
05:13 PM
"I tried adding the licence again."
Once it's expired it's expired. Licences run for a fixed period. They cannot be re-used.
... View more
06-03-2016
05:11 PM
It's not true to say a paid licence does not expire. It IS possible to purchase a perpetual licence, but they don't come cheap (and they still have an expiry date, just one that's 20 years or so hence). Ordinarily paid licencing expires annually.
... View more
05-09-2016
04:00 PM
I'd be inclined to fire off the alert to a single, collective address, and have the mail server expand it. That way you don't have to maintain the mail list if recipient addresses change. It becomes part of the natural id management of users. It's also just more readily achievable. Keeping your roles in text files seems a strangely archaic way of doing things. Do you not manage authentication, roles, mail groups, etc. with some kind of centralised directory service like LDAP or AD?
On a Linux Splunk server, you could conceivably have a cron job which recreates an app-packaged search/alert config, or simply use an address list as a recipient list in an alert-spawned script-generated e-mail.
... View more
05-09-2016
03:50 PM
Splunk is not designed to run multiple instances. That doesn't mean you can't. But it is unecessary hard work when there are so many ways to just virtualise.
... View more
05-09-2016
01:48 PM
Search and tabulate the entries by punctuation, and see of there's a spike in any particular pattern.
Tabulate by hour, and look for the start of the increase.
Find the source of the anomaly.
The answer to your question is in the very data itself...
One thing to look out for is that you're not doing some weekly rotation and ending up with all the renumbered rotated logs being reindexed. (If you're using log rotation you also need to implement appropriate black/whitelist rules.)
... View more
04-10-2016
03:33 PM
Well, funnily enough, the GeoIP database is exactly what is at fault here. Following up over 2 years on, but here's an article that references exactly this: http://fusion.net/story/287592/internet-mapping-glitch-kansas-farm/
... View more
03-23-2016
05:52 PM
Your better option is to set up syslog-ng or other native syslogd , and use a local file consumer to read the logs generated. ( syslog-ng is by far the best specific implementation, and widely recommended on here.)
There are no shortage of sources of information on the subject: Primary documentation search; Best practices for Configuring Syslog Input (Wiki); Best practices for Configuring Syslog Input (wiki).
But to answer your question strictly UDP 514 is the syslog default (provided you have set up a listening service to capture it).
... View more
03-09-2016
05:34 AM
Indeed it should - duly corrected.
... View more
01-22-2016
05:49 PM
2 Karma
84600 seconds is actually 23:30. I think you mean 86400.
... View more
01-11-2016
10:13 AM
There are methods for removing indexed data from visibility, and then re-indexing. But that would eat into your daily consumption cap. There is no method that I know of for bulk manipulation of extant index data.
... View more
01-11-2016
10:08 AM
1 Karma
I don't follow what you're trying to do. Why do you want to run Splunk's bundled openssl manually?
Splunk bundles all its required binaries and libraries together in the local install, but from shell your LD_LIBRARY_PATH and other loadable library configs are going to be completely out of whack.
If you really mean to use the Splunk-distributed version of openssl rather than a proper local copy, you can specify the path to libs directory, thus:
# LD_LIBRARY_PATH=/opt/splunk/lib/ /opt/splunk/bin/openssl version
... View more
12-11-2015
10:35 AM
Well, again, you're assuming that the Heavy Forwarder is at fault in not forwarding. I'm still suspicious of the UDP service. It could be that the system is dropping it in favour of something else, with the service restart prompting a reset.
... View more
12-11-2015
09:10 AM
My first thought is have you guaranteed that at periods when you think it is not being forwarded it is actually getting there in the first place. UDP is not guaranteed delivery, and may end up dumped on the floor in times of congestion or failure.
You're better off using a proper Syslog sub-service over TCP, and locally ingesting the log files generated. (As it happens, this was also Splunk's official advice last time I looked.) My faithful recommendation is to use syslog-ng for its flexibility of configuration.
... View more
10-11-2015
09:46 AM
That doesn't help you, at least not greatly. The search is still going to appear in the logs when it is executed. It only obscures it from direct view in the UI, so again, any administrator will be able to see it with ease if they choose to go looking. It still doesn't provide a total solution.
... View more
10-08-2015
03:25 PM
OK, well I understand your problem, but regardless of the intent or motivation the reality doesn't change. Regardless of the fact someone didn't like my original answer, the fact remains it can't be done.
You can't do it with file permissions, because Splunk as an entirety runs as the same system user (more often than not with sysadmin rights which will override any permissions anyway), and at the application level a user account with administration privileges has total access to everything within the application.
Short of setting up a dedicated Splunk search head administered by the right people, you simply cannot ring-fence the data.
... View more
10-08-2015
03:19 PM
1 Karma
Well that was a facepalm moment - I forgot the stanza heading...
Now there's another problem.
... View more
10-08-2015
01:00 PM
2 Karma
It would defeat the object of being an administrator if the administrator did not have total access to the system.
It also seems very destructive to refuse to collaborate with co-workers, especially those responsible for a service you are using. If I was the administrator I'd be all the more curious about what it was you had to hide.
And no. An administrator can see everything, if they choose to go looking.
... View more
10-08-2015
12:42 PM
1 Karma
I've quickly skimmed through the answers already here, and not found a corresponding answer, although there is a question from 4 years ago which sort of touches on the subject. I have a server on 6.2.4 (which will go to 6.3.0 as soon as I fix this particular issue).
It has two interfaces, one of which potentially faces the big bad world. That interface is currently shut down, but as a precaution, in case someone enables it again, I want to bind Splunk to the internal interface only. The problem is, that when I set SPLUNK_BINDIP to the internal address (RFC 1918 private address space) although the initial web UI comes up prompting me to download an update, as soon as I go beyond that it gives a server 500 error, with a footnote that it is trying to connect to the daemon on 127.0.0.1:8089, but the daemon too is only bound to the same RFC 1918 address, and not on 127.0.0.1.
1) There seems to be no mechanism for specifying binding to all interfaces/addresses excluding exceptions (in order to disregard the external interface)
2) There seems to be no mechanism for specifying multiple explicit address/interface bindings (in order to explicitly list the internal interface and loopback)
3) Despite setting mgmtHostPort in ~splunk/etc/system/local/web.conf to the bound IP address, btool is showing me that Splunk is still picking it up from ~splunk/etc/system/default/web.conf as 127.0.0.1.
What gives?
(It's running on Debian, and yes, ownership and permission on web.conf are correct - or at least they match those on default. This is a root run process, and all ownerships are by root. Still one of my biggest gripes with Splunk architecture on *ix.)
... View more
10-06-2015
02:32 PM
I would, however, like to be corrected and shown wrong.
... View more
10-06-2015
02:31 PM
You could ask the sysadmins to install tcpdump for you. It's not a definitive answer, but I don't know of any debug mode as such.
... View more
10-01-2015
11:09 AM
There's no mystery. Splunk is using its own embedded Python module to create its own session with your local mail relay, rather than relying on a local MTA. There is nothing stopping you doing the same.
This article addresses the problem specifically on Ubuntu, but the method is completely portable. It may require that you install a system-wide deployment of Python. Trying to use Splunk's embedded Python from a cron job will be a hiding to nothing. Bear in mind though that anything you do to by-pass the need for an MTA is also shooting holes in your security policy, because anything available to your script will be available to an intruder.
... View more
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
06-05-2020
02:03 AM
|
