Skip to main content
Version: v5.0

AWS S3 Requirements

AWS S3 Overview#

Amazon Simple Storage Service (S3) is a scalable storage service offered by AWS that provides object storage through a web service interface.

Setting up AWS S3 for SMI#

To set up AWS S3 for SMI, do the following:

  1. Set up four buckets using your own naming scheme (the names you choose can be configured in the Helm Chart file values.yaml). We suggest using a format such as: ${COMPANY_NAME}-dtplatform-${ENVIRONMENT_SHORTNAME}-${BUCKET_TYPE}-${CLOUD_REGION}
  2. Ensure that the required four buckets are set up as follows:
  • kafka - For the Kafka request reply system (communication between services)
  • filesvc - For the File Service storage
  • scriptmanager - For the Script Manager/Worker log storage
  • datasourcesvc - For the Datasources Service file storage
  1. For example, bucket names for a company called “ACME” in an an environment called prod1 in the us-west-2 region, would be as follows:
  • acme-dtplatform-prod1-kafka-us-west-2
  • acme-dtplatform-prod1-filesvc-us-west-2
  • acme-dtplatform-prod1-scriptmanager-us-west-2
  • acme-dtplatform-prod1-datasourcesvc-us-west-2
  1. Ensure that the following bucket security requirements are met:

  • The buckets should perform server-side encryption.

  • The buckets can enable versioning if required by a customer policy. However, no custom data files are overwritten by design.

  • The buckets should block all public access.

  • The buckets object ownership should have “bucket owner enforced”.

  • The bucket should not allow another account access except as required by the customer security policy.

  • The Cross-Origin Resource Sharing (CORS) policy should be broad unless restrictions are required by the customer’s security policy. See example below.

    [    {        "AllowedHeaders": [            "*"        ],        "AllowedMethods": [            "GET"        ],        "AllowedOrigins": [            "*"        ],        "ExposeHeaders": [],        "MaxAgeSeconds": 1728000    }]

  • The FileService bucket will have a bucket policy to allow the CloudFront Distribution to access objects. Refer to AWS documentation for more information. See sample script below. 

    {  "Version": "2012-10-17",  "Statement": [    {      "Sid": "CloudFrontCanGetObjects",      "Effect": "Allow",      "Principal": {        "Service": "cloudfront.amazonaws.com"      },      "Action": "s3:GetObject",      "Resource": "${FILESVC_BUCKET_ARN}/*",      "Condition": {        "StringEquals": {            "aws:SourceArn": "${CLOUDFRONT_DISTRO_ARN}"        }      }    }  ]}