Run Class

Add immutable properties to the run.

Tags and properties (both dict[str, str]) differ in their mutability. Properties are immutable, so they create a permanent record for auditing purposes. Tags are mutable. For more information about working with tags and properties, see Tag and find runs.

add_properties(properties)

Parameters

Extensibility hook for custom Run types stored in Run History.

static add_type_provider(runtype, run_factory)

Parameters

Mark the run as canceled.

If there is an associated job with a set cancel_uri field, terminate that job as well.

cancel()

Create a child run.

child_run(name=None, run_id=None, outputs=None)

Parameters

Returns

Remarks

This is used to isolate part of a run into a subsection. This can be done for identifiable "parts" of a run that are interesting to separate, or to capture independent metrics across an interation of a subprocess.

If an output directory is set for the child run, the contents of that directory will be uploaded to the child run record when the child is completed.

Remove the files corresponding to the current run on the target specified in the run configuration.

clean()

Returns

Wait for task queue to be processed.

Then run is marked as completed. This is typically used in interactive notebook scenarios.

complete(_set_status=True)

Parameters

Create one or many child runs.

create_children(count=None, tag_key=None, tag_values=None)

Parameters

Returns

Remarks

Either parameter count OR parameters tag_key AND tag_values must be specified.

Download an associated file from storage.

download_file(name, output_file_path=None, _validate_checksum=False)

Parameters

Download files from a given storage prefix (folder name) or the entire container if prefix is unspecified.

download_files(prefix=None, output_directory=None, output_paths=None, batch_size=100, append_prefix=True, timeout_seconds=None)

Parameters

Mark the run as failed.

Optionally set the Error property of the run with a message or exception passed to error_details.

fail(error_details=None, error_code=None, _set_status=True)

Parameters

Wait for task queue to be processed.

flush(timeout_seconds=300)

Parameters

Get the run for this workspace with its run ID.

static get(workspace, run_id)

Parameters

Returns

Download all logs for the run to a directory.

get_all_logs(destination=None)

Parameters

Returns

Get all children for the current run selected by specified filters.

get_children(recursive=False, tags=None, properties=None, type=None, status=None, _rehydrate_runs=True)

Parameters

Returns

Return current service context.

Use this method to retrieve the current service context for logging metrics and uploading files. If allow_offline is True (the default), actions against the Run object will be printed to standard out.

get_context(allow_offline=True, used_for_context_manager=False, **kwargs)

Parameters

Returns

Remarks

This function is commonly used to retrieve the authenticated Run object inside of a script to be submitted for execution via experiment.submit(). This run object is both an authenticated context to communicate with Azure Machine Learning services and a conceptual container within which metrics, files (artifacts), and models are contained.

   run = Run.get_context() # allow_offline=True by default, so can be run locally as well
   ...
   run.log("Accuracy", 0.98)
   run.log_row("Performance", epoch=e, error=err)

Fetch the latest status of the run. If the status of the run is "Queued", it will show the details.

get_detailed_status()

Returns

Remarks

• status: The run's current status. Same value as that returned from get_status().

• details: The detailed information for current status.

   run = experiment.submit(config)
   details = run.get_detailed_status()
   # details = {
   #     'status': 'Queued',
   #     'details': 'Run requested 1 node(s). Run is in pending status.',
   # }

Get the definition, status information, current log files, and other details of the run.

get_details()

Returns

Remarks

The returned dictionary contains the following key-value pairs:

• runId: ID of this run.

• target

• status: The run's current status. Same value as that returned from get_status().

• startTimeUtc: UTC time of when this run was started, in ISO8601.

• endTimeUtc: UTC time of when this run was finished (either Completed or Failed), in ISO8601.

  This key does not exist if the run is still in progress.

• properties: Immutable key-value pairs associated with the run. Default properties include the run's snapshot ID and information about the git repository from which the run was created (if any). Additional properties can be added to a run using add_properties.

• inputDatasets: Input datasets associated with the run.

• outputDatasets: Output datasets associated with the run.

• logFiles

• submittedBy

   run = experiment.start_logging()

   details = run.get_details()
   # details = {
   #     'runId': '5c24aa28-6e4a-4572-96a0-fb522d26fe2d',
   #     'target': 'sdk',
   #     'status': 'Running',
   #     'startTimeUtc': '2019-01-01T13:08:01.713777Z',
   #     'endTimeUtc': '2019-01-01T17:15:65.986253Z',
   #     'properties': {
   #         'azureml.git.repository_uri': 'https://example.com/my/git/repo',
   #         'azureml.git.branch': 'master',
   #         'azureml.git.commit': '7dc972657c2168927a02c3bc2b161e0f370365d7',
   #         'azureml.git.dirty': 'True',
   #         'mlflow.source.git.repoURL': 'https://example.com/my/git/repo',
   #         'mlflow.source.git.branch': 'master',
   #         'mlflow.source.git.commit': '7dc972657c2168927a02c3bc2b161e0f370365d7',
   #         'ContentSnapshotId': 'b4689489-ce2f-4db5-b6d7-6ad11e77079c'
   #     },
   #     'inputDatasets': [{
   #         'dataset': {'id': 'cdebf245-701d-4a68-8055-41f9cf44f298'},
   #         'consumptionDetails': {
   #             'type': 'RunInput',
   #             'inputName': 'training-data',
   #             'mechanism': 'Mount',
   #             'pathOnCompute': '/mnt/datasets/train'
   #         }
   #     }],
   #     'outputDatasets': [{
   #         'dataset': {'id': 'd04e8a19-1caa-4b1f-b318-4cbff9af9615'},
   #         'outputType': 'RunOutput',
   #         'outputDetails': {
   #             'outputName': 'training-result'
   #         }
   #     }],
   #     'runDefinition': {},
   #     'logFiles': {},
   #     'submittedBy': 'Alan Turing'
   # }

Return run status including log file content.

get_details_with_logs()

Returns

Get the environment definition that was used by this run.

get_environment()

Returns

List the files that are stored in association with the run.

get_file_names()

Returns

Retrieve the metrics logged to the run.

If recursive is True (False by default), then fetch metrics for runs in the given run's subtree.

get_metrics(name=None, recursive=False, run_type=None, populate=False)

Parameters

Returns

Remarks

   run = experiment.start_logging() # run id: 123
   run.log("A", 1)
   with run.child_run() as child: # run id: 456
       child.log("A", 2)

   metrics = run.get_metrics()
   # metrics = { 'A': 1 }

   metrics = run.get_metrics(recursive=True)
   # metrics = { '123': { 'A': 1 }, '456': { 'A': 2 } } note key is runId

Fetch the latest properties of the run from the service.

get_properties()

Returns

Remarks

Properties are immutable system-generated information such as duration, date of execution, user, and custom properties added with the add_properties method. For more information, see Tag and find runs.

When submitting a job to Azure Machine Learning, if source files are stored in a local git repository then information about the repo is stored as properties. These git properties are added when creating a run or calling Experiment.submit. For more information about git properties see Git integration for Azure Machine Learning.

Get the secret value from the context of a run.

Get the secret value for the name provided. The secret name references a value stored in Azure Key Vault associated with your workspace. For an example of working with secrets, see Use secrets in training runs.

get_secret(name)

Parameters

Returns

Get the secret values for a given list of secret names.

Get a dictionary of found and not found secrets for the list of names provided. Each secret name references a value stored in Azure Key Vault associated with your workspace. For an example of working with secrets, see Use secrets in training runs.

get_secrets(secrets)

Parameters

Returns

Get the latest snapshot ID.

get_snapshot_id()

Returns

Fetch the latest status of the run.

Common values returned include "Running", "Completed", and "Failed".

get_status()

Returns

Remarks

• NotStarted - This is a temporary state client-side Run objects are in before cloud submission.

• Starting - The Run has started being processed in the cloud. The caller has a run ID at this point.

• Provisioning - Returned when on-demand compute is being created for a given job submission.

• Preparing - The run environment is being prepared:

  • docker image build

  • conda environment setup

• Queued - The job is queued in the compute target. For example, in BatchAI the job is in queued state

  while waiting for all the requested nodes to be ready.

• Running - The job started to run in the compute target.

• Finalizing - User code has completed and the run is in post-processing stages.

• CancelRequested - Cancellation has been requested for the job.

• Completed - The run completed successfully. This includes both the user code and run

  post-processing stages.

• Failed - The run failed. Usually the Error property on a run will provide details as to why.

• Canceled - Follows a cancellation request and indicates that the run is now successfully cancelled.

• NotResponding - For runs that have Heartbeats enabled, no heartbeat has been recently sent.

   run = experiment.submit(config)
   while run.get_status() not in ['Completed', 'Failed']: # For example purposes only, not exhaustive
       print('Run {} not in terminal state'.format(run.id))
       time.sleep(10)

DEPRECATED. Use get_context.

Get the submitted run for this experiment.

get_submitted_run(**kwargs)

Returns

Fetch the latest set of mutable tags on the run from the service.

get_tags()

Returns

Get a list of runs in an experiment specified by optional filters.

static list(experiment, type=None, tags=None, properties=None, status=None, include_children=False, _rehydrate_runs=True)

Parameters

Returns

Remarks

The following code example shows some uses of the list method.

   favorite_completed_runs = Run.list(experiment, status='Completed', tags='favorite')

   all_distinct_runs = Run.list(experiment)
   and_their_children = Run.list(experiment, include_children=True)

   only_script_runs = Run.list(experiment, type=ScriptRun.RUN_TYPE)

Get a list of runs in a compute specified by optional filters.

static list_by_compute(compute, type=None, tags=None, properties=None, status=None)

Parameters

Returns

Log a metric value to the run with the given name.

log(name, value, description='', step=None)

Parameters

Remarks

Logging a metric to a run causes that metric to be stored in the run record in the experiment. You can log the same metric multiple times within a run, the result being considered a vector of that metric. If step is specified for a metric it must be specified for all values.

Log an accuracy table to the artifact store.

The accuracy table metric is a multi-use, non-scalar metric that can be used to produce multiple types of line charts that vary continuously over the space of predicted probabilities. Examples of these charts are ROC, precision-recall, and lift curves.

The calculation of the accuracy table is similar to the calculation of an ROC curve. An ROC curve stores true positive rates and false positive rates at many different probability thresholds. The accuracy table stores the raw number of true positives, false positives, true negatives, and false negatives at many probability thresholds.

There are two methods used for selecting thresholds: "probability" and "percentile." They differ in how they sample from the space of predicted probabilities.

Probability thresholds are uniformly spaced thresholds between 0 and 1. If NUM_POINTS is 5 the probability thresholds would be [0.0, 0.25, 0.5, 0.75, 1.0].

Percentile thresholds are spaced according to the distribution of predicted probabilities. Each threshold corresponds to the percentile of the data at a probability threshold. For example, if NUM_POINTS is 5, then the first threshold would be at the 0th percentile, the second at the 25th percentile, the third at the 50th, and so on.

The probability tables and percentile tables are both 3D lists where the first dimension represents the class label, the second dimension represents the sample at one threshold (scales with NUM_POINTS), and the third dimension always has 4 values: TP, FP, TN, FN, and always in that order.

The confusion values (TP, FP, TN, FN) are computed with the one vs. rest strategy. See the following link for more details: https://en.wikipedia.org/wiki/Multiclass_classification

N = # of samples in validation dataset (200 in example) M = # thresholds = # samples taken from the probability space (5 in example) C = # classes in full dataset (3 in example)

Some invariants of the accuracy table:

• TP + FP + TN + FN = N for all thresholds for all classes
• TP + FN is the same at all thresholds for any class
• TN + FP is the same at all thresholds for any class
• Probability tables and percentile tables have shape [C, M, 4]

Note: M can be any value and controls the resolution of the charts This is independent of the dataset, is defined when calculating metrics, and trades off storage space, computation time, and resolution.

Class labels should be strings, confusion values should be integers, and thresholds should be floats.

log_accuracy_table(name, value, description='')

Parameters

Remarks

Example of a valid JSON value:

   {
       "schema_type": "accuracy_table",
       "schema_version": "1.0.1",
       "data": {
           "probability_tables": [
               [
                   [82, 118, 0, 0],
                   [75, 31, 87, 7],
                   [66, 9, 109, 16],
                   [46, 2, 116, 36],
                   [0, 0, 118, 82]
               ],
               [
                   [60, 140, 0, 0],
                   [56, 20, 120, 4],
                   [47, 4, 136, 13],
                   [28, 0, 140, 32],
                   [0, 0, 140, 60]
               ],
               [
                   [58, 142, 0, 0],
                   [53, 29, 113, 5],
                   [40, 10, 132, 18],
                   [24, 1, 141, 34],
                   [0, 0, 142, 58]
               ]
           ],
           "percentile_tables": [
               [
                   [82, 118, 0, 0],
                   [82, 67, 51, 0],
                   [75, 26, 92, 7],
                   [48, 3, 115, 34],
                   [3, 0, 118, 79]
               ],
               [
                   [60, 140, 0, 0],
                   [60, 89, 51, 0],
                   [60, 41, 99, 0],
                   [46, 5, 135, 14],
                   [3, 0, 140, 57]
               ],
               [
                   [58, 142, 0, 0],
                   [56, 93, 49, 2],
                   [54, 47, 95, 4],
                   [41, 10, 132, 17],
                   [3, 0, 142, 55]
               ]
           ],
           "probability_thresholds": [0.0, 0.25, 0.5, 0.75, 1.0],
           "percentile_thresholds": [0.0, 0.01, 0.24, 0.98, 1.0],
           "class_labels": ["0", "1", "2"]
       }
   }

Log a confusion matrix to the artifact store.

This logs a wrapper around the sklearn confusion matrix. The metric data contains the class labels and a 2D list for the matrix itself. See the following link for more details on how the metric is computed: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html

log_confusion_matrix(name, value, description='')

Parameters

Remarks

Example of a valid JSON value:

   {
       "schema_type": "confusion_matrix",
       "schema_version": "1.0.0",
       "data": {
           "class_labels": ["0", "1", "2", "3"],
           "matrix": [
               [3, 0, 1, 0],
               [0, 1, 0, 1],
               [0, 0, 1, 0],
               [0, 0, 0, 1]
           ]
       }
   }

Log an image metric to the run record.

log_image(name, path=None, plot=None, description='')

Parameters

Remarks

Use this method to log an image file or a matplotlib plot to the run. These images will be visible and comparable in the run record.

Log a list of metric values to the run with the given name.

log_list(name, value, description='')

Parameters

Log predictions to the artifact store.

This logs a metric score that can be used to compare the distributions of true target values to the distribution of predicted values for a regression task.

The predictions are binned and standard deviations are calculated for error bars on a line chart.

log_predictions(name, value, description='')

Parameters

Remarks

Example of a valid JSON value:

   {
       "schema_type": "predictions",
       "schema_version": "1.0.0",
       "data": {
           "bin_averages": [0.25, 0.75],
           "bin_errors": [0.013, 0.042],
           "bin_counts": [56, 34],
           "bin_edges": [0.0, 0.5, 1.0]
       }
   }

Log residuals to the artifact store.

This logs the data needed to display a histogram of residuals for a regression task. The residuals are predicted - actual.

There should be one more edge than the number of counts. Please see the numpy histogram documentation for examples of using counts and edges to represent a histogram. https://docs.scipy.org/doc/numpy/reference/generated/numpy.histogram.html

log_residuals(name, value, description='')

Parameters

Remarks

Example of a valid JSON value:

   {
       "schema_type": "residuals",
       "schema_version": "1.0.0",
       "data": {
           "bin_edges": [50, 100, 200, 300, 350],
           "bin_counts": [0.88, 20, 30, 50.99]
       }
   }

Log a row metric to the run with the given name.

log_row(name, description=None, **kwargs)

Parameters

Remarks

Using log_row creates a table metric with columns as described in kwargs. Each named parameter generates a column with the value specified. log_row can be be called once to log an arbitrary tuple, or multiple times in a loop to generate a complete table.

   citrus = ['orange', 'lemon', 'lime']
   sizes = [ 10, 7, 3]
   for index in range(len(citrus)):
       run.log_row("citrus", fruit = citrus[index], size=sizes[index])

Log a table metric to the run with the given name.

log_table(name, value, description='')

Parameters

Register a model for operationalization.

register_model(model_name, model_path=None, tags=None, properties=None, model_framework=None, model_framework_version=None, description=None, datasets=None, sample_input_dataset=None, sample_output_dataset=None, resource_configuration=None, **kwargs)

Parameters

Returns

Remarks

   model = best_run.register_model(model_name = 'best_model', model_path = 'outputs/model.pkl')

Delete the list of mutable tags on this run.

remove_tags(tags)

Parameters

Returns

Restore a snapshot as a ZIP file. Returns the path to the ZIP.

restore_snapshot(snapshot_id=None, path=None)

Parameters

Returns

Add or modify a set of tags on the run. Tags not passed in the dictionary are left untouched.

You can also add simple string tags. When these tags appear in the tag dictionary as keys, they have a value of None. For more information, see Tag and find runs.

set_tags(tags)

Parameters

Mark the run as started.

This is typically used in advanced scenarios when the run has been created by another actor.

start()

Submit an experiment and return the active child run.

submit_child(config, tags=None, **kwargs)

Parameters

Returns

Remarks

Submit is an asynchronous call to the Azure Machine Learning platform to execute a trial on local or remote hardware. Depending on the configuration, submit will automatically prepare your execution environments, execute your code, and capture your source code and results into the experiment's run history.

To submit an experiment you first need to create a configuration object describing how the experiment is to be run. The configuration depends on the type of trial required.

An example of how to submit a child experiment from your local machine using ScriptRunConfig is as follows:

   from azureml.core import ScriptRunConfig

   # run a trial from the train.py code in your current directory
   config = ScriptRunConfig(source_directory='.', script='train.py',
       run_config=RunConfiguration())
   run = parent_run.submit_child(config)

   # get the url to view the progress of the experiment and then wait
   # until the trial is complete
   print(run.get_portal_url())
   run.wait_for_completion()

For details on how to configure a run, see submit.

Tag the run with a string key and optional string value.

tag(key, value=None)

Parameters

Remarks

Tags and properties on a run are both dictionaries of string -> string. The difference between them is mutability: Tags can be set, updated, and deleted while Properties can only be added. This makes Properties more appropriate for system/workflow related behavior triggers, while Tags are generally user-facing and meaningful for the consumers of the experiment.

   run = experiment.start_logging()
   run.tag('DeploymentCandidate')
   run.tag('modifiedBy', 'Master CI')
   run.tag('modifiedBy', 'release pipeline') # Careful, tags are mutable

   run.add_properties({'BuildId': os.environ.get('VSTS_BUILD_ID')}) # Properties are not

   tags = run.get_tags()
   # tags = { 'DeploymentCandidate': None, 'modifiedBy': 'release pipeline' }

Save a snapshot of the input file or folder.

take_snapshot(file_or_folder_path)

Parameters

Returns

Remarks

Snapshots are intended to be the source code used to execute the experiment run. These are stored with the run so that the run trial can be replicated in the future.

Upload a file to the run record.

upload_file(name, path_or_stream, datastore_name=None)

Parameters

Returns

Remarks

   run = experiment.start_logging()
   run.upload_file(name='important_file', path_or_stream="path/on/disk/file.txt")

Upload files to the run record.

upload_files(names, paths, return_artifacts=False, timeout_seconds=None, datastore_name=None)

Parameters

Remarks

upload_files has the same effect as upload_file on separate files, however there are performance and resource utilization benefits when using upload_files.

   import os

   run = experiment.start_logging()
   file_name_1 = 'important_file_1'
   file_name_2 = 'important_file_2'
   run.upload_files(names=[file_name_1, file_name_2],
                       paths=['path/on/disk/file_1.txt', 'other/path/on/disk/file_2.txt'])

   run.download_file(file_name_1, 'file_1.txt')

   os.mkdir("path")  # The path must exist
   run.download_file(file_name_2, 'path/file_2.txt')

Upload the specified folder to the given prefix name.

upload_folder(name, path, datastore_name=None)

Parameters

Remarks

   run = experiment.start_logging()
   run.upload_folder(name='important_files', path='path/on/disk')

   run.download_file('important_files/existing_file.txt', 'local_file.txt')

Wait for the completion of this run. Returns the status object after the wait.

wait_for_completion(show_output=False, wait_post_processing=False, raise_on_error=True)

Parameters

Returns

Return the run description.

The optional description of the run is a user-specified string useful for describing a run.

Returns

Return the run display name.

The optional display name of the run is a user-specified string useful for later identification of the run.

Returns

Get experiment containing the run.

Returns

Get run ID.

The ID of the run is an identifier unique across the containing experiment.

Returns

DEPRECATED. Use display_name.

The optional name of the run is a user-specified string useful for later identification of the run.

Returns

Get run number.

A monotonically increasing number representing the order of runs within an experiment.

Returns

Fetch the parent run for this run from the service.

Runs can have an optional parent, resulting in a potential tree hierarchy of runs. To log metrics to a parent run, use the log method of the parent object, for example, run.parent.log().

Returns

Return the immutable properties of this run.

Returns

Remarks

Properties include immutable system-generated information such as duration, date of execution, user, etc.

Return the run object's status.

Return the set of mutable tags on this run.

Returns

Get run type.

Indicates how the run was created or configured.

Returns