Synapse Multipart Upload

Implements the client side of Synapse multipart upload, which provides a robust means of uploading large files (into the 10s of GB). End users should not need to call any of these functions directly.

synapseclient.multipart_upload.calculate_part_size(fileSize, partSize=None, min_part_size=8388608, max_parts=10000)

Parts for multipart upload must be at least 5 MB and there must be at most 10,000 parts

synapseclient.multipart_upload.count_completed_parts(part_status)

Given a string of the form “1001110”, where 1 and 0 indicate a status of completed or not, return the count of parts already completed.

synapseclient.multipart_upload.find_parts_to_upload(part_status)

Given a string of the form “1001110”, where 1 and 0 indicate a status of completed or not, return the part numbers that aren’t completed.

synapseclient.multipart_upload.get_data_chunk(data, n, chunksize=8388608)

Return the nth chunk of a buffer.

synapseclient.multipart_upload.get_file_chunk(filepath, n, chunksize=8388608)

Read the nth chunk from the file.

synapseclient.multipart_upload.multipart_upload(syn, filepath, filename=None, contentType=None, storageLocationId=None, **kwargs)

Upload a file to a Synapse upload destination in chunks.

Parameters:
  • syn – a Synapse object
  • filepath – the file to upload
  • filename – upload as a different filename
  • contentType – contentType
  • partSize – number of bytes per part. Minimum 5MB.
  • storageLocationId – a id indicating where the file should be stored. Retrieved from Synapse’s UploadDestination
Returns:

a File Handle ID

Keyword arguments are passed down to _multipart_upload() and _start_multipart_upload().

synapseclient.multipart_upload.multipart_upload_string(syn, text, filename=None, contentType=None, storageLocationId=None, **kwargs)

Upload a string using the multipart file upload.

Parameters:
  • syn – a Synapse object
  • text – a string to upload as a file.
  • filename – a string containing the base filename
  • contentType – contentType
  • partSize – number of bytes per part. Minimum 5MB.
  • storageLocationId – a id indicating where the text should be stored. Retrieved from Synapse’s UploadDestination
Returns:

a File Handle ID

Keyword arguments are passed down to _multipart_upload() and _start_multipart_upload().

synapseclient.multipart_upload._multipart_upload(syn, filename, contentType, get_chunk_function, md5, fileSize, partSize=None, storageLocationId=None, **kwargs)

Multipart Upload.

Parameters:
  • syn – a Synapse object
  • filename – a string containing the base filename
  • contentType – contentType
  • get_chunk_function – a function that takes a part number and size and returns the bytes of that chunk of the file
  • md5 – the part’s MD5 as hex.
  • fileSize – total number of bytes
  • partSize – number of bytes per part. Minimum 5MB.
  • storageLocationId – a id indicating where the file should be stored. retrieved from Synapse’s UploadDestination
Returns:

a MultipartUploadStatus object

Keyword arguments are passed down to _start_multipart_upload().

synapseclient.multipart_upload._start_multipart_upload(syn, filename, md5, fileSize, partSize, contentType, preview=True, storageLocationId=None, forceRestart=False)
Returns:A MultipartUploadStatus