Local development¶

The dyff.audit.local module contains tools for developing analyses locally without needing to interact with a full Dyff platform instance.

The primary interface for local development is the DyffLocalPlatform class. The local platform implements most of the Dyff API functionality exposed by the Dyff Client with the same interface but running entirely within Python.

There are also mock-ups for inference functionality in dyff.audit.local.mocks. These can be used to simulate running a Dyff dyff.schema.platform.InferenceService without the overhead of running an actual machine learning model.

class dyff.audit.local.DyffLocalPlatform(storage_root: Path | str = 'dyff-outputs', *, remote_client: Client | None = None)¶

Emulates a subset of Dyff Platform operations locally.

This class is intended to aid in local development and debugging of code and data that will run on the Dyff Platform. The inferface mirrors that of dyff.client.Client and should have similar behavior.

Entities created on the local platform are stored in a local cache. You can optionally provide a dyff.client.Client instance, in which case you can use the .fetch() functions to download an entity and all associated artifacts to local storage.

entity_path(id: str) → Path¶: Returns the path to the local directory that stores information about the entity with the given ID.

link_entity(package: Path, id: str | None = None) → str¶

Creates a symlink in the local storage tree pointing to a directory on the file system. You can either provide an ID or let the platform generate one. Returns the assigned ID.

This can be used to create an “editable” package so that you can refer to the package at a stable ID while still changing the contents of the files in it.

Parameters:

package – The path to the directory to symlink to.
id – If provided, becomes the ID of the linked package in the platform. Otherwise, a new ID is generated.

Returns:

The ID of the linked entity.

property account: str¶: A dummy local account name.

property datasets: _Datasets¶: Operations on Dataset entities.

property evaluations: _Evaluations¶: Operations on Evaluation entities.

property inferenceservices: _InferenceServices¶: Operations on InferenceService entities.

property inferencesessions: _InferenceSessions¶: Operations on InferenceSession entities.

property measurements: _Measurements¶: Operations on Measurement entities.

property methods: _Methods¶: Operations on Method entities.

property models: _Models¶: Operations on Model entities.

property modules: _Modules¶: Operations on Module entities.

property remote: Client | None¶: The remote client instance.

property reports: _Reports¶: Operations on Report entities.

property safetycases: _SafetyCases¶: Operations on SafetyCase entities.

property scores: _Scores¶: Operations on Score entities.

property storage_root: Path¶: The root of the local storage directory tree.

class dyff.audit.local.platform._Datasets¶

get(id: str) → Dataset | None¶: Get the database record associated with an entity.

fetch(id: str) → Dataset¶: Fetch an entity and all associated artifacts from the remote Dyff instance associated with the remote client that was provided to the DyffLocalPlatform constructor.

create(request: DatasetCreateRequest | dict[str, Any]) → Dataset¶

Create a new entity.

Parameters:: request – The entity create request specification.
Returns:: A full entity spec with its .id and other system properties set

create_arrow_dataset(dataset_directory: str, *, account: str, name: str) → Dataset¶

Create a Dataset resource describing an existing Arrow dataset.

Internally, constructs a DatasetCreateRequest using information obtained from the Arrow dataset, then calls create() with the constructed request.

Typical usage:

dataset = client.datasets.create_arrow_dataset(dataset_directory, ...)
client.datasets.upload_arrow_dataset(dataset, dataset_directory)

Parameters:

dataset_directory (str) – The root directory of the Arrow dataset.
account (str) – The account that will own the Dataset resource.
name (str) – The name of the Dataset resource.

Returns:

The complete Dataset resource.

Return type:

Dataset

delete(id: str) → None¶: Set the entity’s status to "Deleted".

purge(id: str) → None¶: Remove an entity and all associated artifacts from local storage.

upload_arrow_dataset(dataset: Dataset, dataset_directory: str) → None¶

Uploads the data files in an existing Arrow dataset for which a Dataset resource has already been created.

Typical usage:

dataset = client.datasets.create_arrow_dataset(dataset_directory, ...)
client.datasets.upload_arrow_dataset(dataset, dataset_directory)

Parameters:

dataset (Dataset) – The Dataset resource for the Arrow dataset.
dataset_directory (str) – The root directory of the Arrow dataset.

class dyff.audit.local.platform._Evaluations¶

get(id: str) → Evaluation | None¶: Get the database record associated with an entity.

fetch(id: str) → Evaluation¶: Fetch an entity and all associated artifacts from the remote Dyff instance associated with the remote client that was provided to the DyffLocalPlatform constructor.

create(request: EvaluationCreateRequest | dict[str, Any]) → Evaluation¶

Create a new entity.

Parameters:: request – The entity create request specification.
Returns:: A full entity spec with its .id and other system properties set

delete(id: str) → None¶: Set the entity’s status to "Deleted".

import_data(dataset_directory: Path | str, *, id: str | None = None, evaluation_request: EvaluationCreateRequest | None = None) → Evaluation¶

Imports existing data into the local cache under a newly-generated ID. Useful when you already have evaluation data and you want to make it available to other workflows using the local platform.

Parameters:: dataset_directory (str) – The root directory of the Arrow dataset.
Returns:: The ID of the imported data in the local platform.
Return type:: str

local_evaluation(*, dataset: str, inferencesession: str) → str¶

Emulate an Evaluation workflow by feeding data from a local Dataset to an InferenceSession running on the Dyff platform.

The output dataset will have the same schema as the outputs from an Evaluation run on the platform, including fields added by the platform – _index_, _replication_, etc.

The input dataset must be compatible with the canonical Dyff Platform dataset schema for the appropriate inference task.

Parameters:

dataset (str) – The ID of a Dataset in local storage.
inferencesession (ID) – The ID of an InferenceSession that is already running on the remote Dyff instance.

Returns:

An ID for the evaluation. This will not correspond to an entity in the the local or remote datastores, but it can be used to derive the IDs of replications in the output dataset.

Return type:

str

purge(id: str) → None¶: Remove an entity and all associated artifacts from local storage.

class dyff.audit.local.platform._InferenceServices¶

get(id: str) → InferenceService | None¶: Get the database record associated with an entity.

fetch(id: str) → InferenceService¶: Fetch an entity and all associated artifacts from the remote Dyff instance associated with the remote client that was provided to the DyffLocalPlatform constructor.

create_mock(mock_type: Type[InferenceServiceMock], *, account: str, id: str | None = None, model: str | None = None) → InferenceService¶

Create an InferenceService that uses a mock-up backend.

The mock_type can be anything that implements the InferenceServiceMock interface and is importable locally; you can implement and use your own mock services.

Parameters:

mock_type – The type of the mock service implementation.
account – The account that should own the InferenceService.
id – Will be generated if not given.
model – The ID of a model to associate with the service. Since this is a mock service, usually the model will also be a mock-up (i.e., the model.source.kind field will be Mock).

Returns:

A full entity spec with its .id and other system properties set

delete(id: str) → None¶: Set the entity’s status to "Deleted".

purge(id: str) → None¶: Remove an entity and all associated artifacts from local storage.

class dyff.audit.local.platform._InferenceSessions¶

get(id: str) → InferenceSession | None¶: Get the database record associated with an entity.

fetch(id: str) → InferenceSession¶: Fetch an entity and all associated artifacts from the remote Dyff instance associated with the remote client that was provided to the DyffLocalPlatform constructor.

client(session_id: str, token: str, *, interface: InferenceInterface | None = None, endpoint: str | None = None, input_adapter: Adapter | None = None, output_adapter: Adapter | None = None) → InferenceSessionClientMock¶

Create a mock-up inference session client.

The token should be one returned either from inferencesessions.create() or from inferencesessions.token(session_id).

The inference endpoint in the session must also be specified, either directly through the endpoint argument or by specifying an interface. Specifying interface will also use the input and output adapters from the interface. You can also specify these separately in the input_adapter and output_adapter. The non-interface arguments override the corresponding values in interface if both are specified.

Parameters:

session_id – The inference session to connect to
token – An access token with permission to run inference against the session. The token is ignored because the local platform does not implement authorization.
interface – The interface to the session. Either interface or endpoint must be specified.
endpoint – The inference endpoint in the session to call. Either endpoint or interface must be specified.
input_adapter – Optional input adapter, applied to the input before sending it to the session. Will override the input adapter from interface if both are specified.
output_adapter – Optional output adapter, applied to the output of the session before returning to the client. Will override the output adapter from interface if both are specified.

Returns:

An mock-up inference client that makes inference calls to the specified session.

create(request: InferenceSessionCreateRequest | dict[str, Any]) → InferenceSessionAndToken¶

Create a new entity.

Parameters:: request – The entity create request specification.
Returns:: A full entity spec with its .id and other system properties set

delete(id: str) → None¶: Set the entity’s status to "Deleted".

infer(id: str, endpoint: str, request: dict[str, Any]) → list[dict[str, Any]]¶

Make an inference call to a mock-up inference session.

Parameters:

id – The session ID
endpoint – The endpoint to call
request – The input data for inference

Returns:

The inference output

purge(id: str) → None¶: Remove an entity and all associated artifacts from local storage.

ready(id: str) → bool¶

Returns True if the session is ready to receive inference requests.

Parameters:: id – The session ID
Returns:: True if the session is ready to receive inference requests.

token(id: str, expires: datetime | None = None) → str¶

Get an auth token allowing inference calls to the session.

The local platform does not implement authorization, so this will be a “dummy” token.

class dyff.audit.local.platform._Measurements¶

get(id: str) → Measurement | None¶: Get the database record associated with an entity.

fetch(id: str) → Measurement¶: Fetch an entity and all associated artifacts from the remote Dyff instance associated with the remote client that was provided to the DyffLocalPlatform constructor.

create(request: AnalysisCreateRequest | dict[str, Any]) → Measurement¶

Create a new entity.

Parameters:: request – The entity create request specification.
Returns:: A full entity spec with its .id and other system properties set

delete(id: str) → None¶: Set the entity’s status to "Deleted".

purge(id: str) → None¶: Remove an entity and all associated artifacts from local storage.

class dyff.audit.local.platform._Methods¶

get(id: str) → Method | None¶: Get the database record associated with an entity.

fetch(id: str) → Method¶: Fetch an entity and all associated artifacts from the remote Dyff instance associated with the remote client that was provided to the DyffLocalPlatform constructor.

create(request: MethodCreateRequest | dict[str, Any]) → Method¶

Create a new entity.

Parameters:: request – The entity create request specification.
Returns:: A full entity spec with its .id and other system properties set

delete(id: str) → None¶: Set the entity’s status to "Deleted".

purge(id: str) → None¶: Remove an entity and all associated artifacts from local storage.

class dyff.audit.local.platform._Models¶

get(id: str) → Model | None¶: Get the database record associated with an entity.

fetch(id: str) → Model¶: Fetch an entity and all associated artifacts from the remote Dyff instance associated with the remote client that was provided to the DyffLocalPlatform constructor.

create(request: ModelCreateRequest | dict[str, Any]) → Model¶

Create a new entity.

Parameters:: request – The entity create request specification.
Returns:: A full entity spec with its .id and other system properties set

delete(id: str) → None¶: Set the entity’s status to "Deleted".

purge(id: str) → None¶: Remove an entity and all associated artifacts from local storage.

class dyff.audit.local.platform._Modules¶

get(id: str) → Module | None¶: Get the database record associated with an entity.

fetch(id: str) → Module¶: Fetch an entity and all associated artifacts from the remote Dyff instance associated with the remote client that was provided to the DyffLocalPlatform constructor.

create(request: ModuleCreateRequest | dict[str, Any]) → Module¶

Create a new entity.

Parameters:: request – The entity create request specification.
Returns:: A full entity spec with its .id and other system properties set

create_package(package_directory: str, *, account: str, name: str) → Module¶

Create a Module resource describing a package structured as a directory tree.

Internally, constructs a ModuleCreateRequest using information obtained from the directory tree, then calls create() with the constructed request.

Typical usage:

module = client.modules.create_package(package_directory, ...)
client.modules.upload_package(module, package_directory)

Parameters:

package_directory (str) – The root directory of the package.
account (str) – The account that will own the Module resource.
name (str) – The name of the Module resource.

Returns:

The complete Module resource.

Return type:

Module

delete(id: str) → None¶: Set the entity’s status to "Deleted".

purge(id: str) → None¶: Remove an entity and all associated artifacts from local storage.

upload_package(module: Module, package_directory: str) → None¶

Uploads the files in a package directory for which a Module resource has already been created.

Typical usage:

module = client.modules.create_package(package_directory, ...)
client.modules.upload_package(module, package_directory)

Parameters:

module (Module) – The Module resource for the package.
package_directory (str) – The root directory of the package.

class dyff.audit.local.platform._Reports¶

get(id: str) → Report | None¶: Get the database record associated with an entity.

fetch(id: str) → Report¶: Fetch an entity and all associated artifacts from the remote Dyff instance associated with the remote client that was provided to the DyffLocalPlatform constructor.

create(request: ReportCreateRequest | dict[str, Any]) → Report¶

Create a new entity.

Parameters:: request – The entity create request specification.
Returns:: A full entity spec with its .id and other system properties set

delete(id: str) → None¶: Set the entity’s status to "Deleted".

purge(id: str) → None¶: Remove an entity and all associated artifacts from local storage.

class dyff.audit.local.platform._SafetyCases¶

get(id: str) → SafetyCase | None¶: Get the database record associated with an entity.

fetch(id: str) → SafetyCase¶: Fetch an entity and all associated artifacts from the remote Dyff instance associated with the remote client that was provided to the DyffLocalPlatform constructor.

create(request: AnalysisCreateRequest | dict[str, Any]) → SafetyCase¶

Create a new entity.

Parameters:: request – The entity create request specification.
Returns:: A full entity spec with its .id and other system properties set

delete(id: str) → None¶: Set the entity’s status to "Deleted".

purge(id: str) → None¶: Remove an entity and all associated artifacts from local storage.

class dyff.audit.local.platform._Scores¶

get(*, analysis: str) → list[Score]¶: Get all the scores output by the given analysis.