sagemaker.core.lineage

sagemaker.core.lineage#

SageMaker Lineage tracking and artifact management.

class sagemaker.core.lineage.Action(sagemaker_session=None, **kwargs)[source]#

Bases: Record

An Amazon SageMaker action, which is part of a SageMaker lineage.

Examples

from sagemaker.lineage import action

my_action = action.Action.create(
    action_name='MyAction',
    action_type='EndpointDeployment',
    source_uri='s3://...')

my_action.properties["added"] = "property"
my_action.save()

for actn in action.Action.list():
    print(actn)

my_action.delete()

action_arn#

The ARN of the action.

Type:: str

action_name#

The name of the action.

Type:: str

action_type#

The type of the action.

Type:: str

description#

A description of the action.

Type:: str

status#

The status of the action.

Type:: str

source#

The source of the action with a URI and type.

Type:: obj

properties#

Dictionary of properties.

Type:: dict

tags#

A list of tags to associate with the action.

Type:: List[dict[str, str]]

creation_time#

When the action was created.

Type:: datetime

created_by#

Contextual info on which account created the action.

Type:: obj

last_modified_time#

When the action was last modified.

Type:: datetime

last_modified_by#

Contextual info on which account created the action.

Type:: obj

action_arn: str = None#

action_name: str = None#

action_type: str = None#

artifacts(direction: LineageQueryDirectionEnum = LineageQueryDirectionEnum.BOTH) → List[Artifact][source]#

Use a lineage query to retrieve all artifacts that use this action.

Parameters:: direction (LineageQueryDirectionEnum, optional) – The query direction.
Returns:: Artifacts.
Return type:: list of Artifacts

Create an action and return an Action object representing it.

Parameters:

action_name (str) – Name of the action
source_uri (str) – Source URI of the action
source_type (str) – Source type of the action
action_type (str) – The type of the action
description (str) – Description of the action
status (str) – Status of the action.
properties (dict) – key/value properties
tags (dict) – AWS tags for the action
sagemaker_session (sagemaker.session.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

A SageMaker Action object.

Return type:

Action

created_by: str = None#

creation_time: datetime = None#

delete(disassociate: bool = False)[source]#

Delete the action.

Parameters:: disassociate (bool) – When set to true, disassociate incoming and outgoing association.

description: str = None#

last_modified_by: str = None#

last_modified_time: datetime = None#

Return a list of action summaries.

Parameters:

source_uri (str, optional) – A source URI.
action_type (str, optional) – An action type.
created_before (datetime.datetime, optional) – Return actions created before this instant.
created_after (datetime.datetime, optional) – Return actions created after this instant.
sort_by (str, optional) – Which property to sort results by. One of ‘SourceArn’, ‘CreatedBefore’, ‘CreatedAfter’
sort_order (str, optional) – One of ‘Ascending’, or ‘Descending’.
max_results (int, optional) – maximum number of actions to retrieve
next_token (str, optional) – token for next page of results
sagemaker_session (sagemaker.session.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

An iterator: over ActionSummary objects.

Return type:

collections.Iterator[ActionSummary]

classmethod load(action_name: str, sagemaker_session=None) → Action[source]#

Load an existing action and return an Action object representing it.

Parameters:

action_name (str) – Name of the action
sagemaker_session (sagemaker.session.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

A SageMaker Action object

Return type:

Action

properties: dict = None#

properties_to_remove: list = None#

save() → Action[source]#

Save the state of this Action to SageMaker.

Returns:: A SageMaker ``Action``object.
Return type:: Action

set_tag(tag=None)[source]#

Add a tag to the object.

Args:

Returns:: str}): a list of key value pairs
Return type:: list({str

set_tags(tags=None)[source]#

Add tags to the object.

Parameters:: tags (Optional[Tags]) – list of key value pairs.
Returns:: str}): a list of key value pairs
Return type:: list({str

source: ActionSource = None#

status: str = None#

tags: list = None#

class sagemaker.core.lineage.Artifact(sagemaker_session=None, **kwargs)[source]#

Bases: Record

An Amazon SageMaker artifact, which is part of a SageMaker lineage.

Examples

from sagemaker.lineage import artifact

my_artifact = artifact.Artifact.create(
    artifact_name='MyArtifact',
    artifact_type='S3File',
    source_uri='s3://...')

my_artifact.properties["added"] = "property"
my_artifact.save()

for artfct in artifact.Artifact.list():
    print(artfct)

my_artifact.delete()

artifact_arn#

The ARN of the artifact.

Type:: str

artifact_name#

The name of the artifact.

Type:: str

artifact_type#

The type of the artifact.

Type:: str

source#

The source of the artifact with a URI and types.

Type:: obj

properties#

Dictionary of properties.

Type:: dict

tags#

A list of tags to associate with the artifact.

Type:: List[dict[str, str]]

creation_time#

When the artifact was created.

Type:: datetime

created_by#

Contextual info on which account created the artifact.

Type:: obj

last_modified_time#

When the artifact was last modified.

Type:: datetime

last_modified_by#

Contextual info on which account created the artifact.

Type:: obj

artifact_arn: str = None#

artifact_name: str = None#

artifact_type: str = None#

Create an artifact and return an Artifact object representing it.

Parameters:

artifact_name (str, optional) – Name of the artifact
source_uri (str, optional) – Source URI of the artifact
source_types (list, optional) – Source types
artifact_type (str, optional) – Type of the artifact
properties (dict, optional) – key/value properties
tags (dict, optional) – AWS tags for the artifact
sagemaker_session (sagemaker.session.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

A SageMaker Artifact object.

Return type:

Artifact

created_by: str = None#

creation_time: datetime = None#

delete(disassociate: bool = False)[source]#

Delete the artifact object.

Parameters:: disassociate (bool) – When set to true, disassociate incoming and outgoing association.

downstream_trials(sagemaker_session=None) → list[source]#

Use the lineage API to retrieve all downstream trials that use this artifact.

Parameters:: sagemaker_session (obj) – Sagemaker Session to use. If not provided a default session will be created.
Returns:: A list of SageMaker Trial objects.
Return type:: [Trial]

downstream_trials_v2() → list[source]#

Use a lineage query to retrieve all downstream trials that use this artifact.

Returns:: A list of SageMaker Trial objects.
Return type:: [Trial]

last_modified_by: str = None#

last_modified_time: datetime = None#

Return a list of artifact summaries.

Parameters:

source_uri (str, optional) – A source URI.
artifact_type (str, optional) – An artifact type.
created_before (datetime.datetime, optional) – Return artifacts created before this instant.
created_after (datetime.datetime, optional) – Return artifacts created after this instant.
sort_by (str, optional) – Which property to sort results by. One of ‘SourceArn’, ‘CreatedBefore’,’CreatedAfter’
sort_order (str, optional) – One of ‘Ascending’, or ‘Descending’.
max_results (int, optional) – maximum number of artifacts to retrieve
next_token (str, optional) – token for next page of results
sagemaker_session (sagemaker.session.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

An iterator: over ArtifactSummary objects.

Return type:

collections.Iterator[ArtifactSummary]

classmethod load(artifact_arn: str, sagemaker_session=None) → Artifact[source]#

Load an existing artifact and return an Artifact object representing it.

Parameters:

artifact_arn (str) – ARN of the artifact
sagemaker_session (sagemaker.session.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

A SageMaker Artifact object

Return type:

Artifact

properties: dict = None#

s3_uri_artifacts(s3_uri: str) → dict[source]#

Retrieve a list of artifacts that use provided s3 uri.

Parameters:: s3_uri (str) – A S3 URI.
Returns:: A list of Artifacts

save() → Artifact[source]#

Save the state of this Artifact to SageMaker.

Note that this method must be run from a SageMaker context such as Studio or a training job due to restrictions on the CreateArtifact API.

Returns:: A SageMaker Artifact object.
Return type:: Artifact

set_tag(tag=None)[source]#

Add a tag to the object.

Parameters:: tag (obj) – Key value pair to set tag.
Returns:: str}): a list of key value pairs
Return type:: list({str

set_tags(tags=None)[source]#

Add tags to the object.

Parameters:: tags (Optional[Tags]) – list of key value pairs.
Returns:: str}): a list of key value pairs
Return type:: list({str

source: ArtifactSource = None#

tags: list = None#

upstream_trials() → List[source]#

Use the lineage query to retrieve all upstream trials that use this artifact.

Returns:: A list of SageMaker Trial objects.
Return type:: [Trial]

class sagemaker.core.lineage.Association(sagemaker_session=None, **kwargs)[source]#

Bases: Record

An Amazon SageMaker artifact, which is part of a SageMaker lineage.

Examples

from sagemaker.lineage import association

my_association = association.Association.create(
    source_arn=artifact_arn,
    destination_arn=trial_component_arn,
    association_type='ContributedTo')

for assoctn in association.Association.list():
    print(assoctn)

my_association.delete()

source_arn#

The ARN of the source entity.

Type:: str

source_type#

The type of the source entity.

Type:: str

destination_arn#

The ARN of the destination entity.

Type:: str

destination_type#

The type of the destination entity.

Type:: str

association_type#

the type of the association.

Type:: str

classmethod create(source_arn: str, destination_arn: str, association_type: str | None = None, sagemaker_session=None) → Association[source]#

Add an association and return an Association object representing it.

Parameters:

source_arn (str) – The ARN of the source.
destination_arn (str) – The ARN of the destination.
association_type (str) – The type of the association. ContributedTo, AssociatedWith, DerivedFrom, or Produced.
sagemaker_session (sagemaker.session.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

A SageMaker Association object.

Return type:

association

delete()[source]#: Delete this Association from SageMaker.

destination_arn: str = None#

Return a list of context summaries.

Parameters:

source_arn (str) – The ARN of the source entity.
destination_arn (str) – The ARN of the destination entity.
source_type (str) – The type of the source entity.
destination_type (str) – The type of the destination entity.
association_type (str) – The type of the association.
created_after (datetime.datetime, optional) – Return contexts created after this instant.
created_before (datetime.datetime, optional) – Return contexts created before this instant.
sort_by (str, optional) – Which property to sort results by. One of ‘SourceArn’, ‘CreatedBefore’, ‘CreatedAfter’
sort_order (str, optional) – One of ‘Ascending’, or ‘Descending’.
max_results (int, optional) – maximum number of contexts to retrieve
next_token (str, optional) – token for next page of results
sagemaker_session (sagemaker.session.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

An iterator: over AssociationSummary objects.

Return type:

collections.Iterator[AssociationSummary]

set_tag(tag=None)[source]#

Add a tag to the object.

Parameters:: tag (obj) – Key value pair to set tag.
Returns:: str}): a list of key value pairs
Return type:: list({str

set_tags(tags=None)[source]#

Add tags to the object.

Parameters:: ([{key (tags) – value}]): list of key value pairs.
Returns:: str}): a list of key value pairs
Return type:: list({str

source_arn: str = None#

class sagemaker.core.lineage.Context(sagemaker_session=None, **kwargs)[source]#

Bases: Record

An Amazon SageMaker context, which is part of a SageMaker lineage.

context_arn#

The ARN of the context.

Type:: str

context_name#

The name of the context.

Type:: str

context_type#

The type of the context.

Type:: str

description#

A description of the context.

Type:: str

source#

The source of the context with a URI and type.

Type:: obj

properties#

Dictionary of properties.

Type:: dict

tags#

A list of tags to associate with the context.

Type:: List[dict[str, str]]

creation_time#

When the context was created.

Type:: datetime

created_by#

Contextual info on which account created the context.

Type:: obj

last_modified_time#

When the context was last modified.

Type:: datetime

last_modified_by#

Contextual info on which account created the context.

Type:: obj

actions(direction: LineageQueryDirectionEnum) → List[Action][source]#

Use the lineage query to retrieve actions that use this context.

Parameters:: direction (LineageQueryDirectionEnum) – The query direction.
Returns:: Actions.
Return type:: list of Actions

context_arn: str = None#

context_name: str = None#

context_type: str = None#

Create a context and return a Context object representing it.

Parameters:

context_name (str) – The name of the context.
source_uri (str) – The source URI of the context.
source_type (str) – The type of the source.
context_type (str) – The type of the context.
description (str) – Description of the context.
properties (dict) – Metadata associated with the context.
tags (dict) – Tags to add to the context.
sagemaker_session (sagemaker.session.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

A SageMaker Context object.

Return type:

Context

created_by: str = None#

creation_time: datetime = None#

delete(disassociate: bool = False)[source]#

Delete the context object.

Parameters:: disassociate (bool) – When set to true, disassociate incoming and outgoing association.
Returns:: boto API response.
Return type:: obj

last_modified_by: str = None#

last_modified_time: datetime = None#

Return a list of context summaries.

Parameters:

source_uri (str, optional) – A source URI.
context_type (str, optional) – An context type.
created_before (datetime.datetime, optional) – Return contexts created before this instant.
created_after (datetime.datetime, optional) – Return contexts created after this instant.
sort_by (str, optional) – Which property to sort results by. One of ‘SourceArn’, ‘CreatedBefore’, ‘CreatedAfter’
sort_order (str, optional) – One of ‘Ascending’, or ‘Descending’.
max_results (int, optional) – maximum number of contexts to retrieve
next_token (str, optional) – token for next page of results
sagemaker_session (sagemaker.session.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

An iterator: over ContextSummary objects.

Return type:

collections.Iterator[ContextSummary]

classmethod load(context_name: str, sagemaker_session=None) → Context[source]#

Load an existing context and return an Context object representing it.

Examples

from sagemaker.lineage import context

my_context = context.Context.create(
    context_name='MyContext',
    context_type='Endpoint',
    source_uri='arn:aws:...')

my_context.properties["added"] = "property"
my_context.save()

for ctx in context.Context.list():
    print(ctx)

my_context.delete()

Args:: context_name (str): Name of the context sagemaker_session (sagemaker.session.Session): Session object which

manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.
Returns:: Context: A SageMaker Context object

properties: dict = None#

save() → Context[source]#

Save the state of this Context to SageMaker.

Returns:: boto API response.
Return type:: obj

set_tag(tag=None)[source]#

Add a tag to the object.

Parameters:: tag (obj) – Key value pair to set tag.
Returns:: str}): a list of key value pairs
Return type:: list({str

set_tags(tags=None)[source]#

Add tags to the object.

Parameters:: ([{key (tags) – value}]): list of key value pairs.
Returns:: str}): a list of key value pairs
Return type:: list({str

tags: list = None#

class sagemaker.core.lineage.LineageEntityEnum(value)[source]#

Bases: Enum

Enum of lineage entities for use in a query filter.

ACTION = 'Action'#

ARTIFACT = 'Artifact'#

CONTEXT = 'Context'#

TRIAL = 'Trial'#

TRIAL_COMPONENT = 'TrialComponent'#

Bases: object

A filter used in a lineage query.

class sagemaker.core.lineage.LineageQuery(sagemaker_session)[source]#

Bases: object

Creates an object used for performing lineage queries.

query(start_arns: List[str], direction: LineageQueryDirectionEnum = LineageQueryDirectionEnum.BOTH, include_edges: bool = True, query_filter: LineageFilter | None = None, max_depth: int = 10) → LineageQueryResult[source]#

Perform a lineage query.

Parameters:

start_arns (List[str]) – A list of ARNs that will be used as the starting point for the query.
direction (LineageQueryDirectionEnum, optional) – The direction of the query.
include_edges (bool, optional) – If true, return edges in addition to vertices.
query_filter (LineageQueryFilter, optional) – The query filter.

Returns:

The lineage query result.

Return type:

LineageQueryResult

class sagemaker.core.lineage.LineageQueryDirectionEnum(value)[source]#

Bases: Enum

Enum of query filter directions.

ASCENDANTS = 'Ascendants'#

BOTH = 'Both'#

DESCENDANTS = 'Descendants'#

class sagemaker.core.lineage.LineageSourceEnum(value)[source]#

Bases: Enum

Enum of lineage types for use in a query filter.

APPROVAL = 'Approval'#

CHECKPOINT = 'Checkpoint'#

DATASET = 'DataSet'#

ENDPOINT = 'Endpoint'#

IMAGE = 'Image'#

MODEL = 'Model'#

MODEL_DATA = 'ModelData'#

MODEL_DEPLOYMENT = 'ModelDeployment'#

MODEL_GROUP = 'ModelGroup'#

MODEL_REPLACE = 'ModelReplaced'#

PROCESSING_JOB = 'ProcessingJob'#

TENSORBOARD = 'TensorBoard'#

TRAINING_JOB = 'TrainingJob'#

TRANSFORM_JOB = 'TransformJob'#

class sagemaker.core.lineage.LineageTableVisualizer(sagemaker_session)[source]#

Bases: object

Creates a dataframe containing the lineage assoociations of a SageMaker object.

Generate a dataframe containing all incoming and outgoing lineage entities.

Examples: .. code-block:: python

viz = LineageTableVisualizer(sagemaker_session) df = viz.show(training_job_name=training_job_name) # in a notebook display(df.to_html())

Parameters:

trial_component_name (str, optional) – Name of a trial component. Defaults to None.
training_job_name (str, optional) – Name of a training job. Defaults to None.
processing_job_name (str, optional) – Name of a processing job. Defaults to None.
pipeline_execution_step (obj, optional) – Pipeline execution step. Defaults to None.
model_package_arn (str, optional) – Model package arn. Defaults to None.
endpoint_arn (str, optional) – Endpoint arn. Defaults to None.
artifact_arn (str, optional) – Artifact arn. Defaults to None.
context_arn (str, optional) – Context arn. Defaults to None.
actions_arn (str, optional) – Action arn. Defaults to None.

Returns:

Pandas dataframe containing lineage associations.

Return type:

DataFrame

class sagemaker.core.lineage.LineageTrialComponent(sagemaker_session=None, **kwargs)[source]#

Bases: Record

An Amazon SageMaker, lineage trial component, which is part of a SageMaker lineage.

A trial component is a stage in a trial. Trial components are created automatically within the SageMaker runtime and also can be created directly. To automatically associate trial components with a trial and experiment supply an experiment config when creating a job. For example: https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateTrainingJob.html

trial_component_name#

The name of the trial component. Generated by SageMaker from the name of the source job with a suffix specific to the type of source job. trial_component_arn (str): The ARN of the trial component.

Type:: str

display_name#

The name of the trial component that will appear in UI, such as SageMaker Studio.

Type:: str

source#

A TrialComponentSource object with a source_arn attribute.

Type:: obj

status#

Status of the source job.

Type:: str

start_time#

When the source job started.

Type:: datetime

end_time#

When the source job ended.

Type:: datetime

creation_time#

When the source job was created.

Type:: datetime

created_by#

Contextual info on which account created the trial component.

Type:: obj

last_modified_time#

When the trial component was last modified.

Type:: datetime

last_modified_by#

Contextual info on which account last modified the trial component.

Type:: obj

parameters#

Dictionary of parameters to the source job.

Type:: dict

input_artifacts#

Dictionary of input artifacts.

Type:: dict

output_artifacts#

Dictionary of output artifacts.

Type:: dict

metrics#

Aggregated metrics for the job.

Type:: obj

parameters_to_remove#

The hyperparameters to remove from the component.

Type:: list

input_artifacts_to_remove#

The input artifacts to remove from the component.

Type:: list

output_artifacts_to_remove#

The output artifacts to remove from the component.

Type:: list

tags#

A list of tags to associate with the trial component.

Type:: List[dict[str, str]]

created_by = None#

creation_time = None#

dataset_artifacts(direction: LineageQueryDirectionEnum = LineageQueryDirectionEnum.ASCENDANTS) → List[Artifact][source]#

Use the lineage query to retrieve datasets that use this trial component.

Parameters:: direction (LineageQueryDirectionEnum, optional) – The query direction.
Returns:: Artifacts representing a dataset.
Return type:: list of Artifacts

display_name = None#

end_time = None#

input_artifacts = None#

input_artifacts_to_remove = None#

last_modified_by = None#

last_modified_time = None#

classmethod load(trial_component_name: str, sagemaker_session=None) → LineageTrialComponent[source]#

Load an existing trial component and return an TrialComponent object representing it.

Parameters:

trial_component_name (str) – Name of the trial component
sagemaker_session (sagemaker.session.Session) – Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created using the default AWS configuration chain.

Returns:

A SageMaker LineageTrialComponent object

Return type:

LineageTrialComponent

metrics = None#

models(direction: LineageQueryDirectionEnum = LineageQueryDirectionEnum.DESCENDANTS) → List[Artifact][source]#

Use the lineage query to retrieve models that use this trial component.

Parameters:: direction (LineageQueryDirectionEnum, optional) – The query direction.
Returns:: Artifacts representing a dataset.
Return type:: list of Artifacts

output_artifacts = None#

output_artifacts_to_remove = None#

parameters = None#

parameters_to_remove = None#

pipeline_execution_arn() → str[source]#

Get the ARN for the pipeline execution associated with this trial component (if any).

Returns:: A pipeline execution ARN.
Return type:: str

source = None#

start_time = None#

status = None#

tags = None#

trial_component_arn = None#

trial_component_name = None#

Modules

`action`	This module contains code to create and manage SageMaker `Actions`.
`artifact`	This module contains code to create and manage SageMaker `Artifact`.
`association`	This module contains code to create and manage SageMaker `Artifact`.
`context`	This module contains code to create and manage SageMaker `Context`.
`lineage_trial_component`	This module contains code to create and manage SageMaker `LineageTrialComponent`.
`query`	This module contains code to query SageMaker lineage.
`visualizer`	This module contains functionality to display lineage data.

sagemaker.core.lineage

Contents

sagemaker.core.lineage#