Challenges and Tasks

A Challenge is a collection of assessments on which participating teams compete to achieve the best performance.

Creating a Challenge

Use the challenges.create() function to create a new challenge:

from dyff.schema.requests import ChallengeCreateRequest
from dyff.schema.platform import ChallengeContent, ChallengeContentPage

challenge = dyffapi.challenges.create(
    ChallengeCreateRequest(
        account=account,
        content=ChallengeContent(
            page=ChallengeContentPage(
                title="My Challenge",
                summary="A challenge to evaluate AI systems",
                body="Detailed description in Markdown format...",
            )
        ),
    )
)

Challenge Tasks

A ChallengeTask is a task that is part of a challenge. Teams make submissions to individual tasks, rather than the overall challenge. A task combines an assessment pipeline that implements the evaluation computations along with rules for submissions and descriptive content for the Web app.

Tasks are embedded within a Challenge and accessed via the Challenge.tasks dictionary.

Creating a Task

Use challenges.create_task() to add a task to an existing challenge:

from dyff.schema.requests import ChallengeTaskCreateRequest
from dyff.schema.platform import (
    ChallengeTaskRules,
    ChallengeTaskExecutionEnvironmentChoices,
    ChallengeTaskExecutionEnvironment,
)

task = dyffapi.challenges.create_task(
    challenge.id,
    ChallengeTaskCreateRequest(
        account=account,
        challenge=challenge.id,
        name="task-1",
        assessment=pipeline.id,  # ID of the Pipeline to assess submissions
        rules=ChallengeTaskRules(
            executionEnvironment=ChallengeTaskExecutionEnvironmentChoices(
                choices={
                    "default": ChallengeTaskExecutionEnvironment(
                        cpu="4",
                        memory="16Gi",
                        accelerators={},
                    )
                }
            )
        ),
    ),
)

Task Rules

Each task defines rules that control submissions:

• Execution Environment: Specifies available computational resources (CPU, memory, accelerators) that submissions can request

• Schedule: Controls when submissions are accepted (opening time, closing time, submission limits per cycle)

Edit task rules with challenges.edit_task_rules():

from dyff.schema.requests import ChallengeTaskRulesEditRequest

dyffapi.challenges.edit_task_rules(
    challenge.id,
    task.id,
    ChallengeTaskRulesEditRequest(
        # ... rule updates
    ),
)

Editing Content

Challenge and task content can be updated after creation:

from dyff.schema.requests import ChallengeContentEditRequest
from dyff.schema.commands import ChallengeContentPatch, ChallengeContentPagePatch

# Edit challenge content
dyffapi.challenges.edit_content(
    challenge.id,
    ChallengeContentEditRequest(
        content=ChallengeContentPatch(
            page=ChallengeContentPagePatch(
                title="Updated Challenge Title",
            )
        )
    ),
)

# Edit task content
dyffapi.challenges.edit_task_content(
    challenge.id,
    task.id,
    ChallengeContentEditRequest(
        content=ChallengeContentPatch(
            page=ChallengeContentPagePatch(
                title="Updated Task Title",
            )
        )
    ),
)