Evaluations

An evaluation object represents a collection of Synapse Entities that will be processed in a particular way. This could mean scoring Entries in a challenge or executing a processing pipeline.

Imports:

from synapseclient import Evaluation, Submission, SubmissionStatus

Evaluations can be retrieved by ID:

evaluation = syn.getEvaluation(1901877)

Like entities, evaluations are access controlled via ACLs. The synapseclient.Synapse.getPermissions() and synapseclient.Synapse.setPermissions() methods work for evaluations:

access = syn.getPermissions(evaluation, user_id)

The synapseclient.Synapse.submit() method returns a Submission object:

entity = syn.get(synapse_id)
submission = syn.submit(evaluation, entity, name='My Data', team='My Team')

The Submission object can then be used to check the status of the submission:

status = syn.getSubmissionStatus(submission)
The status of a submission may be:
  • INVALID the submitted entity is in the wrong format
  • SCORED in the context of a challenge or competition
  • OPEN indicating processing has not completed
  • CLOSED indicating processing has completed

Submission status objects can be updated, usually by changing the status and score fields, and stored back to Synapse using synapseclient.Synapse.store():

status.score = 0.99
status.status = 'SCORED'
status = syn.store(status)

See:

Evaluation

class synapseclient.evaluation.Evaluation(**kwargs)

Keeps track of an evaluation queue in Synapse. Allowing for submissions, retrieval and scoring.

Parameters:
  • name – Name of the evaluation
  • description – A short description describing the evaluation
  • contentSource – Synapse Project or entity associated with the evaluation
  • submissionReceiptMessage – Message to display to users upon submission
  • submissionInstructionsMessage – Message to display to users detailing acceptable formatting for submissions

To create an Evaluation and store it in Synapse:

evaluation = syn.store(Evaluation(
    name="Q1 Final",
    description="Predict progression of MMSE scores for final scoring",
    contentSource="syn2290704"))

The contentSource field links the evaluation to its synapseclient.entity.Project. (Or, really, any synapse ID, but sticking to projects is a good idea.)

Evaluations can be retrieved from Synapse by ID:

evaluation = syn.getEvaluation(1901877)

...by the Synapse ID of the content source (associated entity):

evaluation = syn.getEvaluationByContentSource('syn12345')

...or by the name of the evaluation:

evaluation = syn.getEvaluationByName('Foo Challenge Question 1')

Submission

class synapseclient.evaluation.Submission(**kwargs)

Builds an Synapse submission object.

Parameters:
  • name – Name of submission
  • entityId – Synapse ID of the Entity to submit
  • evaluationId – ID of the Evaluation to which the Entity is to be submitted
  • versionNumber – Version number of the submitted Entity
  • submitterAlias – A pseudonym or team name for a challenge entry

Submission Status

class synapseclient.evaluation.SubmissionStatus(**kwargs)

Builds an Synapse submission status object.

Parameters:
  • score – The score of the submission
  • status – Status can be one of {‘OPEN’, ‘CLOSED’, ‘SCORED’, ‘INVALID’}.