Bulk requests

Dyff supports usage patterns like, “run my analysis on all LLMs in the system.” Accounts have associated resource quotas. You can enqueue as many workflows at once as you want. Workflows that exceed the quota will wait in the status {"status": "Created", "reason": "QuotaLimit"} until resources are available.

Dyff also understands data dependencies between workflows. For example, you can enqueue an Evaluation, and then immediately enqueue a SafetyCase that depends on the output of the Evaluation. The SafetyCase workflow will wait in the status {"status": "Created", "reason": "UnsatisfiedDependency"} until the Evaluation is complete. Queued workflows will immediately fail if one of their dependencies fails.

Example: Repeat an analysis on multiple input datasets

datasets = dyffapi.datasets.query(account="my-account", labels={"source": "reddit.com"})
for dataset in datasets:
    evaluation = dyffapi.evaluations.create(
        EvaluationCreateRequest(
            account="my-account",
            dataset=dataset.id,
            inferenceSession=EvaluationInferenceSessionRequest(
                inferenceService="some-service",
            ),
            replications=10,
            workersPerReplica=10,
        )
    )
    safetycase = dyffapi.safetycases.create(
        AnalysisCreateRequest(
            account="my-account",
            method="my-method",
            inputs=[
                AnalysisInput(keyword="evaluation", entity=evaluation.id),
            ],
        )
    )