AWS S3 Requirements

AWS S3 Overview

Amazon Simple Storage Service (S3) is a scalable storage service offered by AWS that provides object storage through a web service interface.

Setting up AWS S3 for SMI

To set up AWS S3 for SMI, do the following:

1. Set up four buckets using your own naming scheme (the names you choose can be configured in the Helm Chart file values.yaml). We suggest using a format such as: ${COMPANY_NAME}-dtplatform-${ENVIRONMENT_SHORTNAME}-${BUCKET_TYPE}-${CLOUD_REGION}
2. Ensure that the required four buckets are set up as follows:

• kafka - For the Kafka request reply system (communication between services)
• filesvc - For the File Service storage
• scriptmanager - For the Script Manager/Worker log storage
• datasourcesvc - For the Datasources Service file storage

3. For example, bucket names for a company called “ACME” in an an environment called prod1 in the us-west-2 region, would be as follows:

• acme-dtplatform-prod1-kafka-us-west-2
• acme-dtplatform-prod1-filesvc-us-west-2
• acme-dtplatform-prod1-scriptmanager-us-west-2
• acme-dtplatform-prod1-datasourcesvc-us-west-2

4. Ensure that the following bucket security requirements are met:

• The buckets should perform server-side encryption.
• The buckets can enable versioning if required by a customer policy. However, no custom data files are overwritten by design.
• The buckets should block all public access.
• The buckets object ownership should have “bucket owner enforced”.
• The bucket should not allow another account access except as required by the customer security policy.
• The Cross-Origin Resource Sharing (CORS) policy should be broad unless restrictions are required by the customer’s security policy. See example below.

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 1728000
    }
]

• The FileService bucket will have a bucket policy to allow the CloudFront Distribution to access objects. Refer to AWS documentation for more information. See sample script below.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CloudFrontCanGetObjects",
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudfront.amazonaws.com"
      },
      "Action": "s3:GetObject",
      "Resource": "${FILESVC_BUCKET_ARN}/*",
      "Condition": {
        "StringEquals": {
            "aws:SourceArn": "${CLOUDFRONT_DISTRO_ARN}"
        }
      }
    }
  ]
}