sagemaker.core.s3.utils

This module contains helper functions related to S3. You may want to use s3.py instead.

This has a subset of the functions available through s3.py. This module was initially created with functions that were originally in s3.py so that those functions could be imported inside session.py without circular dependencies. (s3.py imports Session as a dependency.)

Functions

determine_bucket_and_prefix([bucket, ...])	Helper function that returns the correct S3 bucket and prefix to use depending on the inputs.
is_s3_url(url)	Returns True if url is an s3 url, False if not
parse_s3_url(url)	Returns an (s3 bucket, key name/prefix) tuple from a url with an s3 scheme.
s3_path_join(*args[, with_end_slash])	Returns the arguments joined by a slash ("/"), similar to os.path.join() (on Unix).

sagemaker.core.s3.utils.determine_bucket_and_prefix(bucket: str | None = None, key_prefix: str | None = None, sagemaker_session=None)

Helper function that returns the correct S3 bucket and prefix to use depending on the inputs.

Parameters:

• bucket (Optional[str]) – S3 Bucket to use (if it exists)

• key_prefix (Optional[str]) – S3 Object Key Prefix to use or append to (if it exists)

• sagemaker_session (sagemaker.session.Session) – Session to fetch a default bucket and prefix from, if bucket doesn’t exist. Expected to exist

Returns: The correct S3 Bucket and S3 Object Key Prefix that should be used

sagemaker.core.s3.utils.is_s3_url(url)

Returns True if url is an s3 url, False if not

Parameters:

url (str)

Return type:

bool

sagemaker.core.s3.utils.parse_s3_url(url)

Returns an (s3 bucket, key name/prefix) tuple from a url with an s3 scheme.

Parameters:

url (str)

Returns:

A tuple containing:

• str: S3 bucket name

• str: S3 key

Return type:

tuple

sagemaker.core.s3.utils.s3_path_join(*args, with_end_slash: bool = False)

Returns the arguments joined by a slash (“/”), similar to os.path.join() (on Unix).

Behavior of this function: - If the first argument is “s3://”, then that is preserved. - The output by default will have no slashes at the beginning or end. There is one exception

(see with_end_slash). For example, s3_path_join(“/foo”, “bar/”) will yield “foo/bar” and s3_path_join(“foo”, “bar”, with_end_slash=True) will yield “foo/bar/”

• Any repeat slashes will be removed in the output (except for “s3://” if provided at the

  beginning). For example, s3_path_join(“s3://”, “//foo/”, “/bar///baz”) will yield “s3://foo/bar/baz”.

• Empty or None arguments will be skipped. For example

  s3_path_join(“foo”, “”, None, “bar”) will yield “foo/bar”

Alternatives to this function that are NOT recommended for S3 paths: - os.path.join(…) will have different behavior on Unix machines vs non-Unix machines - pathlib.PurePosixPath(…) will apply potentially unintended simplification of single

dots (“.”) and root directories. (for example pathlib.PurePosixPath(“foo”, “/bar/./”, “baz”) would yield “/bar/baz”)

• “{}/{}/{}”.format(…) and similar may result in unintended repeat slashes

Parameters:

• *args – The strings to join with a slash.

• with_end_slash (bool) – (default: False) If true and if the path is not empty, appends a “/” to the end of the path

Returns:

The joined string, without a slash at the end unless with_end_slash is True.

Return type:

str