Core

This section is for super users / developers only. These functions are subject to change as they are internal development functions. Use at your own risk.

Multipart Upload

Implements the client side of Synapse multipart upload, which provides a robust means of uploading large files (into the 10s of GB). End users should not need to call any of these functions directly.

synapseclient.core.upload.multipart_upload.multipart_upload_file(syn, file_path: str, dest_file_name: Optional[str] = None, content_type: Optional[str] = None, part_size: Optional[int] = None, storage_location_id: Optional[str] = None, preview: bool = True, force_restart: bool = False, max_threads: Optional[int] = None) → str

Upload a file to a Synapse upload destination in chunks.

Parameters

  • syn – a Synapse object

  • file_path – the file to upload

  • dest_file_name – upload as a different filename

  • content_type – contentType

  • part_size – Number of bytes per part. Minimum 5MB.

  • storage_location_id – an id indicating where the file should be stored. Retrieved from Synapse’s UploadDestination

  • preview – True to generate a preview

  • force_restart – True to restart a previously initiated upload from scratch, False to try to resume

  • max_threads – number of concurrent threads to devote to upload

Returns

a File Handle ID

Keyword arguments are passed down to _multipart_upload() and _start_multipart_upload(). .. _contentType: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.17

synapseclient.core.upload.multipart_upload.multipart_upload_string(syn, text: str, dest_file_name: Optional[str] = None, part_size: Optional[int] = None, content_type: Optional[str] = None, storage_location_id: Optional[str] = None, preview: bool = True, force_restart: bool = False, max_threads: Optional[int] = None)

Upload a file to a Synapse upload destination in chunks.

Parameters

  • syn – a Synapse object

  • text – a string to upload as a file.

  • dest_file_name – upload as a different filename

  • content_type – contentType

  • part_size – number of bytes per part. Minimum 5MB.

  • storage_location_id – an id indicating where the file should be stored. Retrieved from Synapse’s UploadDestination

  • preview – True to generate a preview

  • force_restart – True to restart a previously initiated upload from scratch, False to try to resume

  • max_threads – number of concurrent threads to devote to upload

Returns

a File Handle ID

Keyword arguments are passed down to _multipart_upload() and _start_multipart_upload().

synapseclient.core.upload.multipart_upload.shared_executor(executor)

An outside process that will eventually trigger an upload through the this module can configure a shared Executor by running its code within this context manager.

synapseclient.core.upload.multipart_upload._multipart_upload(syn, dest_file_name, upload_request, part_fn, md5_fn, force_restart: bool = False, max_threads: Optional[int] = None)

Utils

Utility functions useful in the implementation and testing of the Synapse client.

synapseclient.core.utils.as_url(s)

Tries to convert the input into a proper URL.

synapseclient.core.utils.caller_module_name(current_frame)
Parameters

current_frame – use inspect.currentframe().

Returns

the name of the module calling the function, foo(), in which this calling_module() is invoked. Ignores callers that belong in the same module as foo()

synapseclient.core.utils.concrete_type_of(obj: collections.abc.Mapping)

Return the concrete type of an object representing a Synapse entity. This is meant to operate either against an actual Entity object, or the lighter weight dictionary returned by Synapse#getChildren, both of which are Mappings.

class synapseclient.core.utils.deprecated_keyword_param(keywords, version, reason)

A decorator to use to warn when a keyword parameter from a function has been deprecated and is intended for future removal. Will emit a warning such a keyword is passed.

synapseclient.core.utils.download_file(url, localFilepath=None)

Downloads a remote file.

Parameters

localFilePath – May be None, in which case a temporary file is created

Returns

localFilePath

synapseclient.core.utils.equal_paths(path1, path2)

Compare file paths in a platform neutral way

synapseclient.core.utils.extract_filename(content_disposition_header, default_filename=None)

Extract a filename from an HTTP content-disposition header field.

See this memo and this package for cryptic details.

synapseclient.core.utils.extract_prefix(keys)

Takes a list of strings and extracts a common prefix delimited by a dot, for example:

extract_prefix(["entity.bang", "entity.bar", "entity.bat"])
# returns "entity"
synapseclient.core.utils.extract_synapse_id_from_query(query)

An unfortunate hack to pull the synapse ID out of a table query of the form “select column1, column2 from syn12345 where….” needed to build URLs for table services.

synapseclient.core.utils.extract_user_name(profile)

Extract a displayable user name from a user’s profile

synapseclient.core.utils.extract_zip_file_to_directory(zip_file, zip_entry_name, target_dir)

Extracts a specified file in a zip to the specified directory :param zip_file: an opened zip file. e.g. “with zipfile.ZipFile(zipfilepath) as zip_file:” :param zip_entry_name: the name of the file to be extracted from the zip e.g. folderInsideZipIfAny/fileName.txt :param target_dir: the directory to which the file will be extracted

Returns

full path to the extracted file

synapseclient.core.utils.file_url_to_path(url, verify_exists=False)

Convert a file URL to a path, handling some odd cases around Windows paths.

Parameters

  • url – a file URL

  • verify_exists – If true, return an populated dict only if the resulting file path exists on the local file system.

Returns

a path or None if the URL is not a file URL.

synapseclient.core.utils.find_data_file_handle(bundle)

Return the fileHandle whose ID matches the dataFileHandleId in an entity bundle

synapseclient.core.utils.format_time_interval(seconds)

Format a time interval given in seconds to a readable value, e.g. “5 minutes, 37 seconds”.

synapseclient.core.utils.from_unix_epoch_time(ms)

Returns a Datetime object given milliseconds since midnight Jan 1, 1970.

synapseclient.core.utils.from_unix_epoch_time_secs(secs)

Returns a Datetime object given milliseconds since midnight Jan 1, 1970.

synapseclient.core.utils.get_properties(entity)

Returns the dictionary of properties of the given Entity.

synapseclient.core.utils.guess_file_name(string)

Tries to derive a filename from an arbitrary string.

synapseclient.core.utils.id_of(obj)

Try to figure out the Synapse ID of the given object.

Parameters

obj – May be a string, Entity object, or dictionary

Returns

The ID or throws an exception

synapseclient.core.utils.is_base64_encoded(input_string)

Return whether the given input string appears to be base64 encoded

synapseclient.core.utils.is_date(dt)

Objects of class datetime.date and datetime.datetime will be recognized as dates

synapseclient.core.utils.is_in_path(id, path)

Determines whether id is in the path as returned from /entity/{id}/path

Parameters

  • id – synapse id string

  • path – object as returned from ‘/entity/{id}/path’

Returns

True or False

synapseclient.core.utils.is_json(content_type)

detect if a content-type is JSON

synapseclient.core.utils.is_same_base_url(url1, url2)

Compares two urls to see if they are the same excluding up to the base path

Parameters

  • url1 – a URL

  • url2 – a second URL

Returns

Boolean

synapseclient.core.utils.is_synapse_id_str(obj)

If the input is a Synapse ID return it, otherwise return None

synapseclient.core.utils.is_url(s)

Return True if the string appears to be a valid URL.

synapseclient.core.utils.itersubclasses(cls, _seen=None)

http://code.activestate.com/recipes/576949/ (r3)

itersubclasses(cls)

Generator over all subclasses of a given class, in depth first order.

>>> list(itersubclasses(int)) == [bool]
True
>>> class A(object): pass
>>> class B(A): pass
>>> class C(A): pass
>>> class D(B,C): pass
>>> class E(D): pass
>>>
>>> for cls in itersubclasses(A):
...     print(cls.__name__)
B
D
E
C
>>> # get ALL (new-style) classes currently defined
>>> [cls.__name__ for cls in itersubclasses(object)]
['type', ...'tuple', ...]
synapseclient.core.utils.make_bogus_binary_file(n=1048576, filepath=None, printprogress=False)

Makes a bogus binary data file for testing. It is the caller’s responsibility to clean up the file when finished.

Parameters

n – How many bytes to write

Returns

The name of the file

synapseclient.core.utils.make_bogus_data_file(n=100, seed=None)

Makes a bogus data file for testing. It is the caller’s responsibility to clean up the file when finished.

Parameters

  • n – How many random floating point numbers to be written into the file, separated by commas

  • seed – Random seed for the random numbers

Returns

The name of the file

synapseclient.core.utils.md5_for_file(filename, block_size=2097152, callback=None)

Calculates the MD5 of the given file. See source.

Parameters

  • filename – The file to read in

  • block_size – How much of the file to read in at once (bytes). Defaults to 2 MB

  • callback – The callback function that help us show loading spinner on terminal. Defaults to None

Returns

The MD5

synapseclient.core.utils.normalize_path(path)

Transforms a path into an absolute path with forward slashes only.

synapseclient.core.utils.normalize_whitespace(s)

Strips the string and replace all whitespace sequences and other non-printable characters with a single space.

synapseclient.core.utils.printTransferProgress(transferred, toBeTransferred, prefix='', postfix='', isBytes=True, dt=None, previouslyTransferred=0)

Prints a progress bar

Parameters

  • transferred – a number of items/bytes completed

  • toBeTransferred – total number of items/bytes when completed

  • prefix – String printed before progress bar

  • postfix – String printed after progress bar

  • isBytes – A boolean indicating whether to convert bytes to kB, MB, GB etc.

  • dt – The time in seconds that has passed since transfer started is used to calculate rate

  • previouslyTransferred – the number of bytes that were already transferred before this transfer began (e.g. someone ctrl+c’d out of an upload and restarted it later)

synapseclient.core.utils.query_limit_and_offset(query, hard_limit=1000)

Extract limit and offset from the end of a query string.

Returns

A triple containing the query with limit and offset removed, the limit at most equal to the hard_limit, and the offset which defaults to 1

synapseclient.core.utils.snake_case(string)

Convert the given string from CamelCase to snake_case

synapseclient.core.utils.threadsafe_generator(f)

A decorator that takes a generator function and makes it thread-safe. See: http://anandology.com/blog/using-iterators-and-generators/

class synapseclient.core.utils.threadsafe_iter(it)

Takes an iterator/generator and makes it thread-safe by serializing call to the next method of given iterator/generator. See: http://anandology.com/blog/using-iterators-and-generators/

synapseclient.core.utils.to_list(value)

Convert the value (an iterable or a scalar value) to a list.

synapseclient.core.utils.to_unix_epoch_time(dt)

Convert either datetime.date or datetime.datetime objects to UNIX time.

synapseclient.core.utils.to_unix_epoch_time_secs(dt)

Convert either datetime.date or datetime.datetime objects to UNIX time.

synapseclient.core.utils.topolgical_sort(graph)

Given a graph in the form of a dictionary returns a sorted list

Adapted from: http://blog.jupo.org/2012/04/06/topological-sorting-acyclic-directed-graphs/

Parameters

graph – a dictionary with values containing lists of keys referencing back into the dictionary

Returns

sorted list of items

synapseclient.core.utils.touch(path, times=None)

Make sure a file exists. Update its access and modified times.

synapseclient.core.utils.unique_filename(path)

Returns a unique path by appending (n) for some number n to the end of the filename.

Versions

Version Functions

Check for latest version and recommend upgrade:

synapseclient.check_for_updates()

Print release notes for installed version of client:

synapseclient.release_notes()
version_check.check_for_updates()

Check for the existence of newer versions of the client, reporting both current release version and development version.

For help installing development versions of the client, see the docs for synapseclient or the README.md.

version_check.release_notes()

Print release notes for the installed version of the client or latest release or development version if version_url is supplied.

Parameters

version_url –

Defaults to None, meaning release notes for the installed version. Alternatives are:

  • synapseclient.version_check._VERSION_URL

  • synapseclient.version_check._DEV_VERSION_URL