Spunk Addon AWS: How to migrate log from aws:s3 to another sourcetype

[saveriobocca]

Hi everyone,

I am currently receiving data / logs via my buckes.

The following logs have been categorized in the sourcetype following: aws: s3

I would like to create a condition or make sure that certain files in my s3 bucket are stored in another sourcetype and applicate a parsing to line.

Example from:

index=main
sourcetype=aws:s3

To:

index=main
sourcetype=s3_logs_customer

Hi write this in on input.conf but not work:

[source::s3://mypath/*_Report_ProdValid_*.csv]
REPORT-s3-logs-customer = s3-logs-customer

[ s3-logs-customer ]
DATETIME_CONFIG=CURRENT
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=csv
KV_MODE=none
category=Structured
description=Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled=false
pulldown_type=true
FIELD_NAMES=id1,id2,id3,id4
FIELD_QUOTE='
FIELD_DELIMITER=,

Could you give me an example of code to insert in the inputs.conf and transform.conf files to achieve my purpose?

Thanks a lot

[scelikok]

Hi @saveriobocca,

My sample was trying to set sourcetype to s3_logs_customer. [s3_logs_customer_override] should be on transforms.conf. I will put the settings separately.

Did you put correct settings to props.conf and transforms.conf?

props.conf

[source::s3://mypath/*_Report_ProdValid_*.csv]
TRANSFORMS-customer_logs = s3_logs_customer_override

[s3_logs_customer]
DATETIME_CONFIG=CURRENT
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=csv
KV_MODE=none
category=Structured
description=Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled=false
pulldown_type=true
FIELD_NAMES=id1,id2,id3,id4
FIELD_QUOTE='
FIELD_DELIMITER=,

transforms.conf

[s3_logs_customer_override]
DATETIME_CONFIG=CURRENT
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)

[scelikok]

Hi @saveriobocca,

You can put below props.conf and transforms.conf files into the same instance with aws input.

props.conf
[source::s3://mypath/*_Report_ProdValid_*.csv]
TRANSFORMS-customer_logs = s3_logs_customer_override

[s3_logs_customer]
DATETIME_CONFIG=CURRENT
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=csv
KV_MODE=none
category=Structured
description=Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled=false
pulldown_type=true
FIELD_NAMES=id1,id2,id3,id4
FIELD_QUOTE='
FIELD_DELIMITER=,

transforms.conf
[s3_logs_customer_override]
REGEX = .
FORMAT = sourcetype:: s3_logs_customer
DEST_KEY = MetaData:Sourcetype

[saveriobocca]

Hi @scelikok , thanks to response me.

I copy your code on my forwarder but when I check a sourcetype on Search Head I don't view a new sourcetype definited: s3_logs_customer

Why I don't view a sourcetype? I must create a new source type manually by web interface?

-----

In additionally how can I specify a different index for storing the sourcetype? For example the starting sourcetype aws: s3 refers to an app having its index.

index = myindex
sourcetype = aws:s3

I want

index = newindex
sourcetype = s3_logs_customer

The code is similar or I must specify an index in another way?

Thanks,

Saverio