Skip to main content
Version: v5.1

AWS S3 Requirements

AWS S3 Overview#

Amazon Simple Storage Service (S3) is a scalable storage service offered by AWS that provides object storage through a web service interface.

Setting up AWS S3#

To set up AWS S3 for a self-hosted installation, do the following:

  1. Set up the buckets using your own naming scheme (the names you choose can be configured in the Helm Chart values.yaml file). We suggest using a format such as: ${COMPANY_NAME}-dtplatform-${ENVIRONMENT_SHORTNAME}-${BUCKET_TYPE}-${CLOUD_REGION}
  2. Ensure that the following buckets are set up:
  • kafka - For the Kafka request/reply system (communication between services)
  • filesvc - For the File Service storage
  • scriptmanager - For the Script Manager/Worker log storage
  • datasourcesvc - For the Datasource Service file storage
  • workflowsvc - For the Workflow Service storage
  1. For example, the bucket names for a company called “ACME” in an environment called prod1 in the us-west-2 region would be:
  • acme-dtplatform-prod1-kafka-us-west-2
  • acme-dtplatform-prod1-filesvc-us-west-2
  • acme-dtplatform-prod1-scriptmanager-us-west-2
  • acme-dtplatform-prod1-datasourcesvc-us-west-2
  • acme-dtplatform-prod1-workflowsvc-us-west-2
  1. Ensure that the following bucket security requirements are met:

  • The buckets should perform server-side encryption.

  • The buckets can enable versioning if required by customer policy. By design, no Platform-written objects are overwritten.

  • The buckets should block all public access.

  • The buckets' object ownership should be set to “bucket owner enforced”.

  • The buckets should not allow access from other accounts except as required by customer security policy.

  • The Cross-Origin Resource Sharing (CORS) policy should be broad unless restrictions are required by customer security policy. See the example below.

    [    {        "AllowedHeaders": [            "*"        ],        "AllowedMethods": [            "GET"        ],        "AllowedOrigins": [            "*"        ],        "ExposeHeaders": [],        "MaxAgeSeconds": 1728000    }]

  • The FileService bucket needs a bucket policy that allows the CloudFront distribution to access objects. Refer to the AWS documentation for more information. See the sample policy below.

    {  "Version": "2012-10-17",  "Statement": [    {      "Sid": "CloudFrontCanGetObjects",      "Effect": "Allow",      "Principal": {        "Service": "cloudfront.amazonaws.com"      },      "Action": "s3:GetObject",      "Resource": "${FILESVC_BUCKET_ARN}/*",      "Condition": {        "StringEquals": {            "aws:SourceArn": "${CLOUDFRONT_DISTRO_ARN}"        }      }    }  ]}