Skip to content

Evaluation

synapseclient.evaluation

Evaluations

An Evaluation object represents a collection of Synapse Entities that will be processed in a particular way. This could mean scoring Entries in a challenge or executing a processing pipeline.

Imports and authentication:

from synapseclient import Evaluation, Submission, SubmissionStatus, Synapse

syn = Synapse()
syn.login()

Evaluations can be retrieved by ID:

evaluation = syn.getEvaluation(1901877)

Like entities, evaluations are access controlled via ACLs. The synapseclient.Synapse.getPermissions and synapseclient.Synapse.setPermissions methods work for evaluations:

access = syn.getPermissions(evaluation, user_id)

The synapseclient.Synapse.submit method returns a Submission object:

entity = syn.get(synapse_id)
submission = syn.submit(evaluation, entity, name='My Data', team='My Team')

The getSubmissionStatus function can then be used to check the SubmissionStatus of the submission:

status = syn.getSubmissionStatus(submission)
The status of a submission may be
  • INVALID the submitted entity is in the wrong format
  • SCORED in the context of a challenge or competition
  • OPEN indicating processing has not completed
  • CLOSED indicating processing has completed

SubmissionStatus objects can be updated, usually by changing the status and score fields, and stored back to Synapse using synapseclient.Synapse.store:

status.score = 0.99
status.status = 'SCORED'
status = syn.store(status)

See:

Classes

Evaluation

Bases: DictObject

An Evaluation Submission queue, allowing submissions, retrieval and scoring.

PARAMETER DESCRIPTION
name

Name of the evaluation
description

A short description of the evaluation
contentSource

Synapse Project associated with the evaluation
submissionReceiptMessage

Message to display to users upon submission
submissionInstructionsMessage

Message to display to users detailing acceptable formatting for submissions.
Create and store an Evaluation

To create an Evaluation and store it in Synapse:

import synapseclient
from synapseclient import Evaluation

## Initialize a Synapse object & authenticate
syn = synapseclient.Synapse()
syn.login()

evaluation = syn.store(Evaluation(
    name="Q1 Final",
    description="Predict progression of MMSE scores for final scoring",
    contentSource="syn2290704"))

The contentSource field links the evaluation to its Project (or, really, any synapse ID, but sticking to projects is a good idea).

Evaluations can be retrieved from Synapse by ID:

evaluation = syn.getEvaluation(1901877)

...by the Synapse ID of the content source (associated entity):

evaluation = syn.getEvaluationByContentSource('syn12345')

...or by the name of the evaluation:

evaluation = syn.getEvaluationByName('Foo Challenge Question 1')
Source code in synapseclient/evaluation.py 
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
class Evaluation(DictObject):
    """
    An Evaluation Submission queue, allowing submissions, retrieval and scoring.

    Arguments:
        name: Name of the evaluation
        description: A short description of the evaluation
        contentSource: Synapse Project associated with the evaluation
        submissionReceiptMessage: Message to display to users upon submission
        submissionInstructionsMessage: Message to display to users detailing acceptable formatting for submissions.

    Example: Create and store an Evaluation
        To create an [Evaluation](https://rest-docs.synapse.org/rest/org/sagebionetworks/evaluation/model/Evaluation.html)
        and store it in Synapse:

            import synapseclient
            from synapseclient import Evaluation

            ## Initialize a Synapse object & authenticate
            syn = synapseclient.Synapse()
            syn.login()

            evaluation = syn.store(Evaluation(
                name="Q1 Final",
                description="Predict progression of MMSE scores for final scoring",
                contentSource="syn2290704"))

    The *contentSource* field links the evaluation to its [Project][synapseclient.entity.Project]
    (or, really, any synapse ID, but sticking to projects is a good idea).

    [Evaluations](https://rest-docs.synapse.org/rest/org/sagebionetworks/evaluation/model/Evaluation.html)
    can be retrieved from Synapse by ID:

        evaluation = syn.getEvaluation(1901877)

    ...by the Synapse ID of the content source (associated entity):

        evaluation = syn.getEvaluationByContentSource('syn12345')

    ...or by the name of the evaluation:

        evaluation = syn.getEvaluationByName('Foo Challenge Question 1')

    """

    @classmethod
    def getByNameURI(cls, name: str):
        quoted_name = urllib_urlparse.quote(name)
        return f"/evaluation/name/{quoted_name}"

    @classmethod
    def getURI(cls, id: Union[str, int]):
        return f"/evaluation/{id}"

    def __init__(self, **kwargs):
        kwargs["contentSource"] = kwargs.get("contentSource", "")
        if not kwargs["contentSource"].startswith(
            "syn"
        ):  # Verify that synapse Id given
            raise ValueError(
                'The "contentSource" parameter must be specified as a Synapse'
                " Entity when creating an Evaluation"
            )
        super(Evaluation, self).__init__(kwargs)

    def postURI(self):
        return "/evaluation"

    def putURI(self):
        return f"/evaluation/{self.id}"

    def deleteURI(self):
        return f"/evaluation/{self.id}"

    def getACLURI(self):
        return f"/evaluation/{self.id}/acl"

    def putACLURI(self):
        return "/evaluation/acl"

Submission

Bases: DictObject

Builds a Synapse submission object.

PARAMETER DESCRIPTION
name

Name of submission
entityId

Synapse ID of the Entity to submit
evaluationId

ID of the Evaluation to which the Entity is to be submitted
versionNumber

Version number of the submitted Entity
submitterAlias

A pseudonym or team name for a challenge entry
Source code in synapseclient/evaluation.py 
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
class Submission(DictObject):
    """
    Builds a Synapse submission object.

    Arguments:
        name: Name of submission
        entityId: Synapse ID of the Entity to submit
        evaluationId: ID of the Evaluation to which the Entity is to be submitted
        versionNumber: Version number of the submitted Entity
        submitterAlias: A pseudonym or team name for a challenge entry
    """

    @classmethod
    def getURI(cls, id: Union[str, int]):
        return f"/evaluation/submission/{id}"

    def __init__(self, **kwargs):
        if not (
            "evaluationId" in kwargs
            and "entityId" in kwargs
            and "versionNumber" in kwargs
        ):
            raise KeyError

        super().__init__(kwargs)

    def postURI(self):
        return f"/evaluation/submission?etag={self.etag}"

    def putURI(self):
        return f"/evaluation/submission/{self.id}"

    def deleteURI(self):
        return f"/evaluation/submission/{self.id}"

SubmissionStatus

Bases: DictObject

Builds an Synapse submission status object. https://rest-docs.synapse.org/rest/org/sagebionetworks/evaluation/model/SubmissionStatus.html

PARAMETER DESCRIPTION
id

Unique immutable Synapse Id of the Submission

TYPE: Union[str, int]

status

Status can be one of https://rest-docs.synapse.org/rest/org/sagebionetworks/evaluation/model/SubmissionStatusEnum.html.
submissionAnnotations

synapseclient.Annotations to store annotations of submission
canCancel

Can this submission be cancelled?
cancelRequested

Has user requested to cancel this submission?
Source code in synapseclient/evaluation.py 
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
class SubmissionStatus(DictObject):
    """
    Builds an Synapse submission status object.
    <https://rest-docs.synapse.org/rest/org/sagebionetworks/evaluation/model/SubmissionStatus.html>

    Arguments:
        id: Unique immutable Synapse Id of the Submission
        status: Status can be one of
                   <https://rest-docs.synapse.org/rest/org/sagebionetworks/evaluation/model/SubmissionStatusEnum.html>.
        submissionAnnotations: synapseclient.Annotations to store annotations of submission
        canCancel: Can this submission be cancelled?
        cancelRequested: Has user requested to cancel this submission?
    """

    @classmethod
    def getURI(cls, id: Union[str, int]):
        return f"/evaluation/submission/{id}/status"

    def __init__(self, id: Union[str, int], etag: str, **kwargs):
        annotations = kwargs.pop("submissionAnnotations", {})
        # If it is synapse annotations, turn into a format
        # that can be worked with otherwise, create
        # synapseclient.Annotations
        submission_annotations = _convert_to_annotation_cls(
            id=id, etag=etag, values=annotations
        )
        # In Python 3, the super(SubmissionStatus, self) call is equivalent to the parameterless super()
        super().__init__(
            id=id,
            etag=etag,
            submissionAnnotations=submission_annotations,
            **kwargs,
        )

    # def postURI(self):
    #     return '/evaluation/submission/%s/status' % self.id

    def putURI(self):
        return f"/evaluation/submission/{self.id}/status"

    # def deleteURI(self):
    #     return '/evaluation/submission/%s/status' % self.id

    def json(self, ensure_ascii: bool = True):
        """Overloaded json function, turning submissionAnnotations into
        synapse style annotations.

        Arguments:
            ensure_ascii: (default = True) If false, then the return value can contain non-ASCII
            characters. Otherwise, all such characters are escaped in JSON strings.
        Returns:
            A Synapse-style JSON dictionary of annotations.
        """

        json_dict = self
        # If not synapse annotations, turn them into synapseclient.Annotations
        # must have id and etag to turn into synapse annotations
        if not is_synapse_annotations(self.submissionAnnotations):
            json_dict = self.copy()

            annotations = _convert_to_annotation_cls(
                id=self.id, etag=self.etag, values=self.submissionAnnotations
            )
            # Turn into synapse annotation
            json_dict["submissionAnnotations"] = to_synapse_annotations(annotations)
        return json.dumps(
            json_dict, sort_keys=True, indent=2, ensure_ascii=ensure_ascii
        )
Functions
json(ensure_ascii=True)

Overloaded json function, turning submissionAnnotations into synapse style annotations.

PARAMETER DESCRIPTION
ensure_ascii

(default = True) If false, then the return value can contain non-ASCII

TYPE: bool DEFAULT: True

Returns: A Synapse-style JSON dictionary of annotations.

Source code in synapseclient/evaluation.py 
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
def json(self, ensure_ascii: bool = True):
    """Overloaded json function, turning submissionAnnotations into
    synapse style annotations.

    Arguments:
        ensure_ascii: (default = True) If false, then the return value can contain non-ASCII
        characters. Otherwise, all such characters are escaped in JSON strings.
    Returns:
        A Synapse-style JSON dictionary of annotations.
    """

    json_dict = self
    # If not synapse annotations, turn them into synapseclient.Annotations
    # must have id and etag to turn into synapse annotations
    if not is_synapse_annotations(self.submissionAnnotations):
        json_dict = self.copy()

        annotations = _convert_to_annotation_cls(
            id=self.id, etag=self.etag, values=self.submissionAnnotations
        )
        # Turn into synapse annotation
        json_dict["submissionAnnotations"] = to_synapse_annotations(annotations)
    return json.dumps(
        json_dict, sort_keys=True, indent=2, ensure_ascii=ensure_ascii
    )

Functions