topic Re: Splunk App/Add-on for AWS: How to modify aws:s3 sourcetype? in All Apps and Add-ons

Splunk App/Add-on for AWS: How to modify aws:s3 sourcetype?

cwyse — Fri, 13 Nov 2015 18:24:54 GMT

So we are using the aws add on to retrieve elb logs from a s3 bucket. The logs are simply 1 event per a line. But splunk is having trouble indexing them. So the events look something like this:

Svl
ES256-SHA TLSv1
--- "-"
18 HTTP/1.1" "WidgetSystem/6.1.3" AES256-SHA TLSv1

I tried to create a sourcetype to just take things as one line, but when I change sourcetype = aws:s3 to whatever I call my sourcetypes, all the logs just stop working until I change it back. Is there a way to modify the aws:s3 sourcetype to take items as one event per log. Or at least create a new sourcetype I can modify that will keep s3 logs flowing.

Re: Splunk App/Add-on for AWS: How to modify aws:s3 sourcetype?

jcoates_splunk — Sun, 15 Nov 2015 00:15:01 GMT

you should just be able to set the sourcetype that the data is indexed as, then create props.conf and transforms.conf entries as you like for that sourcetype.

Depending on what you mean by "the logs just stop working"...

If they stop being indexed, your changes might be damaging the AWS Add-on's input configuration so that it can't get the logs from AWS anymore?
If they are still getting indexed, but aren't showing up in the formats and/or dashboards that you're expecting, you might be facing a config error or a precedence problem?
If they are no longer the expected content, but instead are media files containing Surrealist films, you might need to explain that it's a cold world for starving artists and that they'd better get back to work?

Re: Splunk App/Add-on for AWS: How to modify aws:s3 sourcetype?

cwyse — Mon, 16 Nov 2015 18:23:18 GMT

So I tried the props.conf as that is normally how I add sourcetypes. But what I mean by they stop working is that nothing gets indexed. You can see them coming in every minute until I restart the forwarders then it just stops. No new events, even when searching just by sourcetype. So I have a feeling your first bullet point is what is happening. But I would be pretty impressed if somehow my amazon ELBs started sending surrealist films.

Re: Splunk App/Add-on for AWS: How to modify aws:s3 sourcetype?

markconlin — Wed, 17 Aug 2016 02:37:56 GMT

I have the exact same question. I want to define a custom sourcetype to be pulled from s3. Cant seem to make it work.

Re: Splunk App/Add-on for AWS: How to modify aws:s3 sourcetype?

jcoates_splunk — Thu, 15 Sep 2016 17:30:38 GMT

FWIW, I just used the UI and it worked fine.

Re: Splunk App/Add-on for AWS: How to modify aws:s3 sourcetype?

markconlin — Thu, 15 Sep 2016 17:35:01 GMT

You used the UI to add a custom source type to an s3 input and once it indexed data it implemented the rules in your custom sourcetype? Where did you define the custom sourcetype?

Re: Splunk App/Add-on for AWS: How to modify aws:s3 sourcetype?

tomasmoser — Thu, 22 Jun 2017 15:30:17 GMT

I have the same problem. Overall configuration S3 input with AWS add-on is magic. Sometimes it works sometimes not. For sure once I change default sourcetype for S3 input from aws:s3 to cisco:umbrella:s3 (my name I chose) the input stops working. Nothing gets indexed.

Having sourcetype named aws:s3 does not make sense. It's not sourcetype, it's basically a source.
Any help?

Tomas