The Data Manager API

A module providing simplified assess to the Squonk2 Data Manager API.

The API URL

The URL to the Data Manager API is taken from the environment variable SQUONK2_DMAPI_URL if it exists. If you do not want to verify the SSL certificate you can also set SQUONK2_DMAPI_VERIFY_SSL_CERT to anything other than "yes", its default value.

If you haven’t set the API URL variables you need to set the Data Manager API URL before you can use any API method.

url = 'https://example.com/data-manager-api'
DmApi.set_api_url(url)

If the Data Manager API is not secure (e.g. you’re developing locally) you can disable the automatic SSL authentication when you set the URL.

dm_api.set_api_url(url, verify_ssl_cert=False)

The API

Python utilities to simplify calls to some parts of the Data Manager API that interact with Projects, Instances (Jobs) and Files.

Note

The URL to the DM API is automatically picked up from the environment variable SQUONK2_DMAPI_URL, expected to be of the form https://example.com/data-manager-api. If the variable isn’t set the user must set it programmatically using DmApi.set_api_url().

class squonk2.dm_api.DmApi

The DmAPI class provides high-level, simplified access to the DM REST API. You can use the request module directly for finer control. This module provides a wrapper around the handling of the request, returning a simplified namedtuple response value DmApiRv

classmethod add_project_editor(access_token: str, *, project_id: str, editor: str, timeout_s: int = 4) → DmApiRv

Adds a user to a Project as an Editor.

Parameters

access_token – A valid DM API access token.
project_id – The Project UUID.
editor – The username to add.
timeout_s – The API request timeout

classmethod add_project_observer(access_token: str, *, project_id: str, observer: str, timeout_s: int = 4) → DmApiRv

Adds a user to a Project as an Observer.

Parameters

access_token – A valid DM API access token.
project_id – The Project UUID.
editor – The username to add.
timeout_s – The API request timeout

classmethod create_project(access_token: str, *, project_name: str, as_tier_product_id: str, private: bool = False, timeout_s: int = 4) → DmApiRv

Creates a Project, which requires a name and a Product ID (a Data Manger Project Tier Product) obtained from the Account Server.

Parameters

access_token – A valid DM API access token.
project_name – A unique name.
as_tier_product_id – If no account server is attached any suitable value can be used. If you are an admin user you can also use the reserved value of product-11111111-1111-1111-1111-111111111111 which is automatically accepted.
timeout_s – The API request timeout

classmethod delete_instance(access_token: str, *, instance_id: str, timeout_s: int = 4) → DmApiRv

Deletes an Instance (Application or Job).

When instances are deleted the container is removed along with the instance-specific directory that is automatically created in the root of the project. Any files in the instance-specific directory will be removed.

Parameters

access_token – A valid DM API access token
instance_id – The instance to delete
timeout_s – The underlying request timeout

classmethod delete_instance_token(*, instance_id: str, token: str, timeout_s: int = 4) → DmApiRv

Deletes a DM API Instance callback token. This API method is not authenticated and therefore does not need an access token. Once the token is deleted no further calls to DmApi.get_unmanaged_project_file_with_token() will be possible. Once deleted the token cannot be re-instantiated.

Parameters

instance_id – A valid DM API instance
token – The callback Token associated with the instance
timeout_s – The API request timeout

classmethod delete_project(access_token: str, *, project_id: str, timeout_s: int = 4) → DmApiRv

Deletes a project.

Parameters

access_token – A valid DM API access token
project_id – The DM-API project id to delete
timeout_s – The underlying request timeout

classmethod delete_unmanaged_project_files(access_token: str, *, project_id: str, project_files: Union[str, List[str]], project_path: str = '/', timeout_s: int = 4) → DmApiRv

Deletes an unmanaged project file, or list of files, on a project path.

Parameters

access_token – A valid DM API access token
project_id – The project where the files are present
project_files – A file or list of files. Leading paths are stripped so the two file files ['dir/file-a.txt', 'file-b.txt'] would be expected to be in the same project directory, i.e. appearing as /file-a.txt and /file-b.txt in the project
project_path – The path in the project where the files are located. The path is relative to the project root and must begin /
timeout_s – The underlying request timeout

classmethod dry_run_job_instance(access_token: str, *, project_id: str, name: str, specification: Dict[str, Any], callback_url: Optional[str] = None, callback_token: Optional[str] = None, callback_context: Optional[str] = None, generate_callback_token: bool = False, debug: Optional[str] = None, timeout_s: int = 4) → DmApiRv

Optionally used prior to starting a Job instance, this method checks that Job Instance can be started in a Project, returning the Job command.

Parameters

access_token – A valid DM API access token
project_id – The project where the files are present
name – A name to associate with the Job
specification – The Job specification, it must contain keys that define the Job’s collection, job name and version. Job-specific variables are passed in using a variables map in the specification
callback_url – An optional URL capable of handling Job callbacks. Must be set if generate_callback_token is used
callback_token – An optional callback token as an alternative to using generate_callback_token
callback_context – An optional context string passed to the callback URL
generate_callback_token – True to instruct the DM to generate a token that can be used with some methods instead of a user access token
debug – Used to prevent the automatic removal of the Job instance. Only use this if you need to
timeout_s – The underlying request timeout

classmethod get_account_server_namespace(access_token: str, *, timeout_s: int = 4) → DmApiRv

Gets Datasets available to the caller.

Parameters

access_token – A valid DM API access token
timeout_s – The underlying request timeout

classmethod get_account_server_registration(access_token: str, *, timeout_s: int = 4) → DmApiRv

Gets Datasets available to the caller.

Parameters

access_token – A valid DM API access token
timeout_s – The underlying request timeout

classmethod get_api_url() → Tuple[str, bool]: Return the API URL and whether validating the SSL layer.

classmethod get_available_datasets(access_token: str, *, timeout_s: int = 4) → DmApiRv

Gets Datasets available to the caller.

Parameters

access_token – A valid DM API access token
timeout_s – The underlying request timeout

classmethod get_available_instances(access_token: str, *, timeout_s: int = 4) → DmApiRv

Gets information about all instances available to you.

Parameters

access_token – A valid DM API access token
timeout_s – The underlying request timeout

classmethod get_available_jobs(access_token: str, *, timeout_s: int = 4) → DmApiRv

Gets a summary list of available Jobs.

Parameters

access_token – A valid DM API access token.
timeout_s – The API request timeout

classmethod get_available_projects(access_token: str, *, project_name: Optional[str] = None, timeout_s: int = 4) → DmApiRv

Gets information about all projects available to you.

Parameters

access_token – A valid DM API access token
project_name – An optional project name to use as a filter
timeout_s – The underlying request timeout

classmethod get_available_tasks(access_token: str, *, exclude_done: bool = False, exclude_purpose: Optional[str] = None, project_id: Optional[str] = None, timeout_s: int = 4) → DmApiRv

Gets information about all tasks available to you.

Parameters

access_token – A valid DM API access token
exclude_done – Set if you want to omit tasks that are ‘done’
exclude_purpose – A comma-separated list of purposes to exclude. Any supported, e.g. DATASET, FILE, INSTANCE, PROJECT
project_id – An optional project ID to limit tasks to
timeout_s – The underlying request timeout

classmethod get_instance(access_token: str, *, instance_id: str, timeout_s: int = 4) → DmApiRv

Gets information about an instance (Application or Job).

Parameters

access_token – A valid DM API access token
instance_id – The specific instance to retrieve
timeout_s – The underlying request timeout

classmethod get_job(access_token: str, *, job_id: int, project_id: Optional[str] = None, timeout_s: int = 4) → DmApiRv

Gets detailed information about a specific Job using the numeric Job record identity.

Parameters

access_token – A valid DM API access token.
job_id – The numeric Job identity
project_id – An optional Project the Job is to be run in, e.g. project-0000
timeout_s – The API request timeout

classmethod get_job_by_version(access_token: str, *, job_collection: str, job_job: str, job_version: str, project_id: str = '', timeout_s: int = 4) → DmApiRv

Gets detailed information about a specific Job using its collection, job and version, using an optional target project_id.

Parameters

access_token – A valid DM API access token.
job_collection – The Job collection, e.g. im-test
job_job – The Job, e.g. nop
job_version – The Job version, e.g. 1.0.0
project_id – An optional Project the Job is to be run in, e.g. project-0000
timeout_s – The API request timeout

classmethod get_job_exchange_rates(access_token: str, *, only_undefined: bool = False, timeout_s: int = 4) → DmApiRv

Gets exchange rates for Jobs.

Parameters

access_token – A valid DM API access token
only_undefined – True to only include jobs that have no exchange rate
timeout_s – The underlying request timeout

classmethod get_jobs(access_token: str, *, project_id: Optional[str] = None, timeout_s: int = 4) → DmApiRv

Gets summary information about all Jobs.

Parameters

access_token – A valid DM API access token.
project_id – An optional Project the Job is to be run in, e.g. project-0000
timeout_s – The API request timeout

classmethod get_project(access_token: str, *, project_id: str, timeout_s: int = 4) → DmApiRv

Gets detailed information about a specific project.

Parameters

access_token – A valid DM API access token
project_id – The specific project to retrieve
timeout_s – The underlying request timeout

classmethod get_project_instances(access_token: str, *, project_id: str, timeout_s: int = 4) → DmApiRv

Gets information about all instances available to you.

Parameters

access_token – A valid DM API access token
project_id – A valid DM project
timeout_s – The underlying request timeout

classmethod get_service_errors(access_token: str, *, include_acknowledged: bool = False, timeout_s: int = 4) → DmApiRv

Gets service errors. You need admin rights to use this method.

Parameters

access_token – A valid DM API access token
include_acknowledged – True to include acknowledged errors
timeout_s – The underlying request timeout

classmethod get_task(access_token: str, *, task_id: str, event_prior_ordinal: int = 0, event_limit: int = 0, timeout_s: int = 4) → DmApiRv

Gets information about a specific Task

Parameters

access_token – A valid DM API access token
task_id – The task
event_prior_ordinal – The event prior ordinal, Use 0 for the first
event_limit – The number of events to return. Use 0 for the default, which depends on the environment, and is typically 500
timeout_s – The underlying request timeout

classmethod get_tasks(access_token: str, *, exclude_done: bool = False, exclude_removal: bool = False, exclude_purpose: Optional[str] = None, project_id: Optional[str] = None, instance_callback_context: Optional[str] = None, timeout_s: int = 4) → DmApiRv

Gets information about a range of Tasks

Parameters

access_token – A valid DM API access token
exclude_done – Set if you do not want to see completed Tasks
exclude_removal – Set if you do not want to see removal Tasks
exclude_purpose – A dot-separated string of purposes to exclude. From INSTANCE, FILE or DATASET
project_id – Limit tasks to the given Project
instance_callback_context – Limit tasks to those for Instances with the given callback context
timeout_s – The underlying request timeout

classmethod get_unmanaged_project_file(access_token: str, *, project_id: str, project_file: str, local_file: str, project_path: str = '/', timeout_s: int = 12) → DmApiRv

Get a single unmanaged file from a project path, save it to the filename defined in local_file.

Parameters

access_token – A valid DM API access token
project_id – The project where the files are present
project_file – The name of the file to get
local_file – The name to use to write the file to on the client
project_path – The path in the project to search for files. The path is relative to the project root and must begin /
timeout_s – The underlying request timeout

classmethod get_unmanaged_project_file_with_token(*, token: str, project_id: str, project_file: str, local_file: str, project_path: str = '/', timeout_s: int = 12) → DmApiRv

Like get_unmanaged_project_file(), this method gets a single unmanaged file from a project path. The method uses an Instance-generated callback token rather than a user-access token.

This method is particularly useful in callback routines where a user access token may not be available. Callback tokens expire and can be deleted, and so this function should only be used when a user access token is not available.

Parameters

token – A DM-generated token, optionally generated when launching instances in the project
project_id – The project where the files are present
project_file – The name of the file to get
local_file – The name to use to write the file to on the client
project_path – The path in the project to search for files. The path is relative to the project root and must begin /
timeout_s – The underlying request timeout

classmethod get_version(access_token: Optional[str] = None, *, timeout_s: int = 4) → DmApiRv

Returns the DM-API service version.

Parameters

access_token – An optional valid DM API access token (deprecated)
timeout_s – The underlying request timeout

classmethod list_project_files(access_token: str, *, project_id: str, project_path: str = '/', include_hidden: bool = False, timeout_s: int = 12) → DmApiRv

Gets a list of project files on a path.

Parameters

access_token – A valid DM API access token
project_id – The project where the files are present
project_path – The path in the project to search for files. The path is relative to the project root and must begin /
include_hidden – Include hidden files in the reponse
timeout_s – The underlying request timeout

classmethod ping(access_token: str, *, timeout_s: int = 4) → DmApiRv

A handy API method that calls the DM API to ensure the server is responding.

Parameters

access_token – A valid DM API access token
timeout_s – The underlying request timeout

classmethod put_job_manifest(access_token: str, *, url: str, header: Optional[str] = None, params: Optional[str] = None, timeout_s: int = 4) → DmApiRv

Puts a Job Manifest onto server. The action requires the token to be that of an admin user.

Parameters

access_token – A valid DM API access token
url – The location of the Manifest (typically a GitHub repository URL)
header – An optional JSON string of header keys and values
params – An optional JSON string of parameter keys and values
timeout_s – The underlying request timeout

classmethod put_unmanaged_project_files(access_token: str, *, project_id: str, project_files: Union[str, List[str]], project_path: str = '/', force: bool = False, timeout_per_file_s: int = 120) → DmApiRv

Puts a file, or list of files, into a DM Project using an optional path.

Parameters

access_token – A valid DM API access token
project_id – The project where the files are to be written
project_files – A file or list of files. Leading paths are stripped so the two file files ['dir/file-a.txt', 'file-b.txt'] would be written to the same project directory, i.e. appearing as /file-a.txt and /file-b.txt in the project
project_path – The path in the project to write the files. The path is relative to the project root and must begin /
force – Files are not written to the project if a file of the same name exists. Here force can be used to over-write files. Files on the server that are immutable cannot be over-written, and doing so will result in an error
timeout_per_file_s – The underlying request timeout

classmethod remove_project_editor(access_token: str, *, project_id: str, editor: str, timeout_s: int = 4) → DmApiRv

Removes a user as an Editor from a Project.

Parameters

access_token – A valid DM API access token.
project_id – The Project UUID.
editor – The username to remove.
timeout_s – The API request timeout

classmethod remove_project_observer(access_token: str, *, project_id: str, observer: str, timeout_s: int = 4) → DmApiRv

Removes a user as an Observer from a Project.

Parameters

access_token – A valid DM API access token.
project_id – The Project UUID.
observer – The username to remove.
timeout_s – The API request timeout

classmethod set_admin_state(access_token: str, *, admin: bool, impersonate: Optional[str] = None, timeout_s: int = 4) → DmApiRv

Adds or removes the become-admin state of your account. Only users whose accounts offer administrative capabilities can use this method.

Parameters

access_token – A valid DM API access token.
admin – True to set admin state
impersonate – An optional username to switch to
timeout_s – The API request timeout

classmethod set_api_url(url: str, *, verify_ssl_cert: bool = True) → None

Replaces the API URL value, which is otherwise set using the SQUONK2_DMAPI_URL environment variable.

Parameters

url – The API endpoint, typically https://example.com/data-manager-api
verify_ssl_cert – Use False to avoid SSL verification in request calls

classmethod set_job_exchange_rates(access_token: str, *, rates: Union[Dict[str, str], List[Dict[str, str]]], timeout_s: int = 4) → DmApiRv

Sets exchange rates for Jobs, given one rate or a list of rates.

A rate is a dictionary with keys ‘collection’, ‘job’, ‘version’, and ‘rate’. An optional ‘comment’ can also be provided. The rate is expected to be a string representation of a decimal, i.e. ‘0.05’.

Parameters

access_token – A valid DM API access token
rates – A rate or a list of rates
timeout_s – The underlying request timeout

classmethod start_job_instance(access_token: str, *, project_id: str, name: str, specification: Dict[str, Any], callback_url: Optional[str] = None, callback_token: Optional[str] = None, callback_context: Optional[str] = None, generate_callback_token: bool = False, debug: Optional[str] = None, timeout_s: int = 4) → DmApiRv

Instantiates a Job Instance in a Project.

Parameters

access_token – A valid DM API access token
project_id – The project where the files are present
name – A name to associate with the Job
specification – The Job specification, it must contain keys that define the Job’s collection, job name and version. Job-specific variables are passed in using a variables map in the specification
callback_url – An optional URL capable of handling Job callbacks. Must be set if generate_callback_token is used
callback_token – An optional callback token as an alternative to using generate_callback_token
callback_context – An optional context string passed to the callback URL
generate_callback_token – True to instruct the DM to generate a token that can be used with some methods instead of a user access token
debug – Used to prevent the automatic removal of the Job instance. Only use this if you need to
timeout_s – The underlying request timeout

class squonk2.dm_api.DmApiRv(success: bool, msg: Dict[Any, Any])

The return value from most of the the DmApi class public methods.

Parameters

success – True if the call was successful, False otherwise.
msg – API request response content

squonk2.dm_api.TEST_PRODUCT_ID: str = 'product-11111111-1111-1111-1111-111111111111': A test Account Server (AS) Product ID. This ID does not actually exist in the AS but is accepted as valid by the Data Manager for Administrative users and used for testing purposes. It allows the creation of Projects without the need of an AS Product.