sagemaker.core.s3

sagemaker.core.s3#

S3 utilities for SageMaker.

class sagemaker.core.s3.S3Downloader[source]#

Bases: object

Contains static methods for downloading directories or files from S3.

static download(s3_uri, local_path, kms_key=None, sagemaker_session=None)[source]#

Static method that downloads a given S3 uri to the local machine.

Parameters:

  • s3_uri (str) – An S3 uri to download from.

  • local_path (str) – A local path to download the file(s) to.

  • kms_key (str) – The KMS key to use to decrypt the files.

  • sagemaker_session (sagemaker.core.helper.session_helper.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

List of local paths of downloaded files

Return type:

list[str]

static list(s3_uri, sagemaker_session=None)[source]#

Static method that lists the contents of an S3 uri.

Parameters:

  • s3_uri (str) – The S3 base uri to list objects in.

  • sagemaker_session (sagemaker.core.helper.session_helper.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

The list of S3 URIs in the given S3 base uri.

Return type:

[str]

static read_bytes(s3_uri, sagemaker_session=None) → bytes[source]#

Static method that returns the contents of a s3 object as bytes.

Parameters:

  • s3_uri (str) – An S3 uri that refers to a s3 object.

  • sagemaker_session (sagemaker.core.helper.session_helper.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

The body of the file.

Return type:

bytes

static read_file(s3_uri, sagemaker_session=None) → str[source]#

Static method that returns the contents of a s3 uri file body as a string.

Parameters:

  • s3_uri (str) – An S3 uri that refers to a single file.

  • sagemaker_session (sagemaker.core.helper.session_helper.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

The body of the file.

Return type:

str

class sagemaker.core.s3.S3Uploader[source]#

Bases: object

Contains static methods for uploading directories or files to S3.

static upload(local_path, desired_s3_uri, kms_key=None, sagemaker_session=None, callback=None)[source]#

Static method that uploads a given file or directory to S3.

Parameters:

  • local_path (str) – Path (absolute or relative) of local file or directory to upload.

  • desired_s3_uri (str) – The desired S3 location to upload to. It is the prefix to which the local filename will be added.

  • kms_key (str) – The KMS key to use to encrypt the files.

  • sagemaker_session (sagemaker.core.helper.session_helper.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

The S3 uri of the uploaded file(s).

static upload_bytes(b: bytes | BytesIO, s3_uri, kms_key=None, sagemaker_session=None)[source]#

Static method that uploads a given file or directory to S3.

Parameters:

  • b (bytes or io.BytesIO) – bytes.

  • s3_uri (str) – The S3 uri to upload to.

  • kms_key (str) – The KMS key to use to encrypt the files.

  • sagemaker_session (sagemaker.core.helper.session_helper.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

The S3 uri of the uploaded file.

Return type:

str

static upload_string_as_file_body(body: str, desired_s3_uri=None, kms_key=None, sagemaker_session=None)[source]#

Static method that uploads a given file or directory to S3.

Parameters:

  • body (str) – String representing the body of the file.

  • desired_s3_uri (str) – The desired S3 uri to upload to.

  • kms_key (str) – The KMS key to use to encrypt the files.

  • sagemaker_session (sagemaker.core.helper.session_helper.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed.

Returns:

The S3 uri of the uploaded file.

Return type:

str

sagemaker.core.s3.determine_bucket_and_prefix(bucket: str | None = None, key_prefix: str | None = None, sagemaker_session=None)[source]#

Helper function that returns the correct S3 bucket and prefix to use depending on the inputs.

Parameters:

  • bucket (Optional[str]) – S3 Bucket to use (if it exists)

  • key_prefix (Optional[str]) – S3 Object Key Prefix to use or append to (if it exists)

  • sagemaker_session (sagemaker.core.helper.session_helper.Session) – Session to fetch a default bucket and prefix from, if bucket doesn’t exist. Expected to exist

Returns: The correct S3 Bucket and S3 Object Key Prefix that should be used

sagemaker.core.s3.is_s3_url(url)[source]#

Returns True if url is an s3 url, False if not

Parameters:

url (str)

Return type:

bool

sagemaker.core.s3.parse_s3_url(url)[source]#

Returns an (s3 bucket, key name/prefix) tuple from a url with an s3 scheme.

Parameters:

url (str)

Returns:

A tuple containing:

  • str: S3 bucket name

  • str: S3 key

Return type:

tuple

sagemaker.core.s3.s3_path_join(*args, with_end_slash: bool = False)[source]#

Returns the arguments joined by a slash (“/”), similar to os.path.join() (on Unix).

Behavior of this function: - If the first argument is “s3://”, then that is preserved. - The output by default will have no slashes at the beginning or end. There is one exception

(see with_end_slash). For example, s3_path_join(“/foo”, “bar/”) will yield “foo/bar” and s3_path_join(“foo”, “bar”, with_end_slash=True) will yield “foo/bar/”

  • Any repeat slashes will be removed in the output (except for “s3://” if provided at the

    beginning). For example, s3_path_join(“s3://”, “//foo/”, “/bar///baz”) will yield “s3://foo/bar/baz”.

  • Empty or None arguments will be skipped. For example

    s3_path_join(“foo”, “”, None, “bar”) will yield “foo/bar”

Alternatives to this function that are NOT recommended for S3 paths: - os.path.join(…) will have different behavior on Unix machines vs non-Unix machines - pathlib.PurePosixPath(…) will apply potentially unintended simplification of single

dots (“.”) and root directories. (for example pathlib.PurePosixPath(“foo”, “/bar/./”, “baz”) would yield “/bar/baz”)

  • “{}/{}/{}”.format(…) and similar may result in unintended repeat slashes

Parameters:

  • *args – The strings to join with a slash.

  • with_end_slash (bool) – (default: False) If true and if the path is not empty, appends a “/” to the end of the path

Returns:

The joined string, without a slash at the end unless with_end_slash is True.

Return type:

str

Modules

client

This module contains Enums and helper methods related to S3.

utils

This module contains helper functions related to S3.
Contents