Run an Evaluation

In Dyff, we call the process of running input data through an AI/ML system to produce outputs an Evaluation. Now that we’ve create a dataset, the next step is to run (or simulate) an Evaluation on that dataset.

Setup

Create an API client as described in the Python client guide:

import os
from dyff.client import Client

dyffapi = Client(api_key=os.environ["DYFF_API_TOKEN"])
ACCOUNT = "<your account ID>"

from dyff.audit.local import DyffLocalPlatform

dyffapi = DyffLocalPlatform(storage_root="/some/dir")
ACCOUNT = "<arbitrary string>"

You’ll also need to know the IDs of a Dataset to use as input and an InferenceService to run on the dataset.

Run an evaluation on the platform

The process of running an evaluation is the same whether you’re using a Client connnected to a Dyff deployment, or a DyffLocalPlatform instance running locally.

Note

Some of the evaluation parameters, such as replicas and useSpotPods, have no effect when using a DyffLocalPlatform instance.

 1from datetime import datetime, timedelta
 2from dyff.schema.requests import (
 3    EvaluationCreateRequest,
 4    EvaluationInferenceSessionRequest,
 5)
 6
 7dyffapi = ...
 8dataset_id: str = ...
 9service_id: str = ...
10evaluation_request = EvaluationCreateRequest(
11    account=account,
12    dataset=dataset_id,
13    inferenceSession=EvaluationInferenceSessionRequest(
14        inferenceService=service_id,
15        expires=datetime.now() + timedelta(days=1),
16        replicas=1,
17        useSpotPods=False,
18    ),
19    replications=2,
20    workersPerReplica=2,
21)
22evaluation = dyffapi.evaluations.create(evaluation_request)
23print(evaluation.json(indent=2))

When running on a Dyff deployment, the create() call will return immediately. You can monitor progress using dyffapi.evaluations.get(evaluation.id).status. The .status will be Complete when the evaluation is finished.

Run local data through a remote session

You can also run an evaluation on local data using a remote inference session. This capability requires that you provide a Client instance that can communicate with an appropriate remotely-hosted Dyff instance:

 1from pathlib import Path
 2
 3from dyff.audit.local import DyffLocalPlatform
 4from dyff.client import Client
 5
 6account = "local"
 7root = Path("/home/me/dyff/my-analysis")
 8
 9dyffremote = Client(...)
10dyfflocal = DyffLocalPlatform(
11    storage_root=root / ".dyff-local", remote_client=dyffremote
12)
13
14dataset = dyfflocal.datasets.create_arrow_dataset(
15    str(root / "arrow_dataset"), account=account, name="test"
16)
17dyfflocal.datasets.upload_arrow_dataset(dataset, str(root / "arrow_dataset"))
18
19inferencesession_id = ...
20evaluation_id = dyfflocal.evaluations.local_evaluation(
21    dataset=dataset.id, inferencesession=inferencesession_id
22)

Then, you can run a local dataset managed by the DyffLocalPlatform through the remote inference session:

 1from pathlib import Path
 2
 3from dyff.audit.local import DyffLocalPlatform
 4from dyff.client import Client
 5
 6account = "local"
 7root = Path("/home/me/dyff/my-analysis")
 8
 9dyffremote = Client(...)
10dyfflocal = DyffLocalPlatform(
11    storage_root=root / ".dyff-local", remote_client=dyffremote
12)
13
14dataset = dyfflocal.datasets.create_arrow_dataset(
15    str(root / "arrow_dataset"), account=account, name="test"
16)
17dyfflocal.datasets.upload_arrow_dataset(dataset, str(root / "arrow_dataset"))
18
19inferencesession_id = ...
20evaluation_id = dyfflocal.evaluations.local_evaluation(
21    dataset=dataset.id, inferencesession=inferencesession_id
22)