Run an Evaluation

In Dyff, we call the process of running input data through an AI/ML system to produce outputs an Evaluation. Now that we’ve create a dataset, the next step is to run (or simulate) an Evaluation on that dataset.

Run an evaluation on the platform

The process of running an evaluation is the same whether you’re using a Client connnected to a Dyff deployment, or a DyffLocalPlatform instance running locally.

Note

Some of the evaluation parameters that control hardware configurations have no effect when using a DyffLocalPlatform instance.

 1from datetime import datetime, timedelta
 2from dyff.schema.requests import (
 3    EvaluationCreateRequest,
 4    EvaluationInferenceSessionRequest,
 5)
 6
 7dyffapi = ...
 8dataset_id: str = ...
 9service_id: str = ...
10evaluation_request = EvaluationCreateRequest(
11    account=account,
12    dataset=dataset_id,
13    inferenceSession=EvaluationInferenceSessionRequest(
14        inferenceService=service_id,
15    ),
16    replications=2,
17)
18evaluation = dyffapi.evaluations.create(evaluation_request)
19print(evaluation.json(indent=2))

This request will run the evaluation with 2 replications, meaning that inference will be run on the entire dataset twice. This capability is useful with non-deterministic inference models to create a distribution over outputs.

When running on a Dyff deployment, the create() call will return immediately. You can monitor progress using dyffapi.evaluations.get(evaluation.id).status. The .status will be Complete when the evaluation is finished.

Run local data through a remote session

You can also run an evaluation on local data using a remote inference session. This capability requires that you provide a Client instance that can communicate with an appropriate remotely-hosted Dyff instance:

 1from pathlib import Path
 2
 3from dyff.audit.local import DyffLocalPlatform
 4from dyff.client import Client
 5
 6account = "local"
 7root = Path("/home/me/dyff/my-analysis")
 8
 9dyffremote = Client(...)
10dyfflocal = DyffLocalPlatform(remote_client=dyffremote)
11
12dataset = dyfflocal.datasets.create_arrow_dataset(
13    str(root / "arrow_dataset"), account=account, name="test"
14)
15dyfflocal.datasets.upload_arrow_dataset(dataset, str(root / "arrow_dataset"))
16
17inferencesession_id = ...
18evaluation_id = dyfflocal.evaluations.local_evaluation(
19    dataset=dataset.id, inferencesession=inferencesession_id
20)

Then, you can run a local dataset managed by the DyffLocalPlatform through the remote inference session:

 1from pathlib import Path
 2
 3from dyff.audit.local import DyffLocalPlatform
 4from dyff.client import Client
 5
 6account = "local"
 7root = Path("/home/me/dyff/my-analysis")
 8
 9dyffremote = Client(...)
10dyfflocal = DyffLocalPlatform(remote_client=dyffremote)
11
12dataset = dyfflocal.datasets.create_arrow_dataset(
13    str(root / "arrow_dataset"), account=account, name="test"
14)
15dyfflocal.datasets.upload_arrow_dataset(dataset, str(root / "arrow_dataset"))
16
17inferencesession_id = ...
18evaluation_id = dyfflocal.evaluations.local_evaluation(
19    dataset=dataset.id, inferencesession=inferencesession_id
20)