I'm using a Splunk Light Cloud instance to index some logs from a web application hosted on Heroku. This is a demo instance that is brand new and has not been customized. I'm using the basic syslog source type to feed a forwarder, which is all working fine.
One thing that surprised me a bit, is that without any configuration every k=v pair is automatically being extracted as a field. e.g. the standard host=xyz dyno=web.1 status=200 entries in the log are each getting extracted to separate fields host , status and dyno . That is all well and good for those fields but currently the size of our Splunk index files reported in the license cube report are 10 times larger than the raw data files and I suspect part of the reason is all the extra fields getting indexed. A lot of our application urls contain k=v type patterns in the query string and these are getting extracted to fields that are not meaningful to us. I'd prefer to remove these fields and just extract the ones I care about using my own regex expressions.
However, I am not able to even determine where this transformation is occurring, much less stop it. From what I can tell, the standard syslog sourcetype has no transformation to perform this extraction. Is there any way to see, given a particular field, what was responsible for its extraction? Or does anyone know specifically how I can prevent these field extractions?
... View more
I just checked again this morning and now the deployment server is accepting connections and my forwarder is showing up in the console. I didn't change anything. So I guess if this happens to anyone in the future, just wait some hours/days ?
... View more
Hi, thanks for your reply. As I mentioned though, I'm using Splunk Light Cloud, so I do not manage or have access to the EC2 rules. From the outside, it looks exactly like a missing rule or wrong security group is applied.
Edit: Sorry just realized you are asking about the forwarder. The forwarder need not listen to 8089, it needs to connect to it on the deployment server. In fact, no host can connect to port 8089 on the deployment server, including my laptop.
... View more
I guess I can't post links. i'm trying to follow instructions on a page you can find by Googling GettingdataintoSplunkLightcloudserviceusingLinux
I am following these instructions to setup a universal forwarder on an EC2 linux host. I am stuck at step 5, perhaps because I cannot determine the correct host and port number for my deployment server, or because it is not running. I thought it would just be input-myinstancename.cloud.splunk.com:8089 but that server is not listening to port 8089. It does listen to 9997 and I tried using that but I think that must be a different service.
These are examples of the errors I see in splunkd.log:
05-24-2019 13:16:54.802 +0000 INFO DC:DeploymentClient - Shutting down phonehome thread.
05-24-2019 13:16:54.803 +0000 INFO DC:DeploymentClient - Closing pubsub connection.
05-24-2019 13:17:45.761 +0000 INFO DC:DeploymentClient - DeploymentClient has been shutdown.
05-24-2019 13:17:45.769 +0000 INFO DC:DeploymentClient - Starting phonehome thread.
05-24-2019 13:17:45.769 +0000 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
05-24-2019 13:17:56.818 +0000 INFO ShutdownHandler - shutting down level "ShutdownLevel_DeploymentClient"
05-24-2019 13:17:56.818 +0000 INFO DC:DeploymentClient - Shutting down phonehome thread.
05-24-2019 13:17:56.818 +0000 INFO DC:DeploymentClient - Closing pubsub connection.
05-24-2019 13:17:56.819 +0000 INFO DC:DeploymentClient - DeploymentClient has been shutdown.
05-24-2019 13:18:00.075 +0000 INFO DC:DeploymentClient - Starting phonehome thread.
05-24-2019 13:18:00.077 +0000 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
05-24-2019 13:18:12.077 +0000 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
... View more