The Data Manager API

A module providing simplified assess to the Squonk2 Data Manager API.

The API URL

The URL to the Data Manager API is taken from the environment variable SQUONK2_DMAPI_URL if it exists. If you do not want to verify the SSL certificate you can also set SQUONK2_DMAPI_VERIFY_SSL_CERT to anything other than "yes", its default value.

If you haven’t set the API URL variables you need to set the Data Manager API URL before you can use any API method.

url = 'https://example.com/data-manager-api'
DmApi.set_api_url(url)

If the Data Manager API is not secure (e.g. you’re developing locally) you can disable the automatic SSL authentication when you set the URL.

DmApi.set_api_url(url, verify_ssl_cert=False)

The API

Python utilities to simplify calls to some parts of the Data Manager API that interact with Projects, Instances (Jobs) and Files.

Note

The URL to the DM API is automatically picked up from the environment variable SQUONK2_DMAPI_URL, expected to be of the form https://example.com/data-manager-api. If the variable isn’t set the user must set it programmatically using DmApi.set_api_url().

class squonk2.dm_api.DmApi

The DmAPI class provides high-level, simplified access to the DM REST API. You can use the request module directly for finer control. This module provides a wrapper around the handling of the request, returning a simplified namedtuple response value DmApiRv

classmethod create_project(access_token: str, *, project_name: str, as_tier_product_id: str, timeout_s: int = 4) → DmApiRv

Creates a Project, which requires a name and a Product ID (a Data Manger Project Tier Product) obtained from the Account Server.

Parameters

  • access_token – A valid DM API access token.

  • project_name – A unique name.

  • as_tier_product_id – If no account server is attached any suitable value can be used. If you are an admin user you can also use the reserved value of product-11111111-1111-1111-1111-111111111111 which is automatically accepted.

  • timeout_s – The API request timeout

classmethod delete_instance(access_token: str, *, instance_id: str, timeout_s: int = 4) → DmApiRv

Deletes an Instance (Application or Job).

When instances are deleted the container is removed along with the instance-specific directory that is automatically created in the root of the project. Any files in the instance-specific directory will be removed.

Parameters

  • access_token – A valid DM API access token

  • instance_id – The instance to delete

  • timeout_s – The underlying request timeout

classmethod delete_instance_token(*, instance_id: str, token: str, timeout_s: int = 4) → DmApiRv

Deletes a DM API Instance callback token. This API method is not authenticated and therefore does not need an access token. Once the token is deleted no further calls to DmApi.get_unmanaged_project_file_with_token() will be possible. Once deleted the token cannot be re-instantiated.

Parameters

  • instance_id – A valid DM API instance

  • token – The callback Token associated with the instance

  • timeout_s – The API request timeout

classmethod delete_project(access_token: str, *, project_id: str, timeout_s: int = 4) → DmApiRv

Deletes a project.

Parameters

  • access_token – A valid DM API access token

  • project_id – The DM-API project id to delete

  • timeout_s – The underlying request timeout

classmethod delete_unmanaged_project_files(access_token: str, *, project_id: str, project_files: Union[str, List[str]], project_path: str = '/', timeout_s: int = 4) → DmApiRv

Deletes an unmanaged project file, or list of files, on a project path.

Parameters

  • access_token – A valid DM API access token

  • project_id – The project where the files are present

  • project_files – A file or list of files. Leading paths are stripped so the two file files ['dir/file-a.txt', 'file-b.txt'] would be expected to be in the same project directory, i.e. appearing as /file-a.txt and /file-b.txt in the project

  • project_path – The path in the project where the files are located. The path is relative to the project root and must begin /

  • timeout_s – The underlying request timeout

classmethod get_account_server_namespace(access_token: str, *, timeout_s: int = 4) → DmApiRv

Gets Datasets available to the caller.

Parameters

  • access_token – A valid DM API access token

  • timeout_s – The underlying request timeout

classmethod get_account_server_registration(access_token: str, *, timeout_s: int = 4) → DmApiRv

Gets Datasets available to the caller.

Parameters

  • access_token – A valid DM API access token

  • timeout_s – The underlying request timeout

classmethod get_api_url() → Tuple[str, bool]

Return the API URL and whether validating the SSL layer.

classmethod get_available_datasets(access_token: str, *, timeout_s: int = 4) → DmApiRv

Gets Datasets available to the caller.

Parameters

  • access_token – A valid DM API access token

  • timeout_s – The underlying request timeout

classmethod get_available_instances(access_token: str, *, timeout_s: int = 4) → DmApiRv

Gets information about all instances available to you.

Parameters

  • access_token – A valid DM API access token

  • timeout_s – The underlying request timeout

classmethod get_available_jobs(access_token: str, *, timeout_s: int = 4) → DmApiRv

Gets a summary list of available Jobs.

Parameters

  • access_token – A valid DM API access token.

  • timeout_s – The API request timeout

classmethod get_available_projects(access_token: str, *, timeout_s: int = 4) → DmApiRv

Gets information about all projects available to you.

Parameters

  • access_token – A valid DM API access token

  • timeout_s – The underlying request timeout

classmethod get_available_tasks(access_token: str, *, exclude_done: bool = False, exclude_purpose: Optional[str] = None, project_id: Optional[str] = None, timeout_s: int = 4) → DmApiRv

Gets information about all tasks available to you.

Parameters

  • access_token – A valid DM API access token

  • exclude_done – Set if you want to omit tasks that are ‘done’

  • exclude_purpose – A comma-separated list of purposes to exclude. Any supported, e.g. DATASET, FILE, INSTANCE, PROJECT

  • project_id – An optional project ID to limit tasks to

  • timeout_s – The underlying request timeout

classmethod get_instance(access_token: str, *, instance_id: str, timeout_s: int = 4) → DmApiRv

Gets information about an instance (Application or Job).

Parameters

  • access_token – A valid DM API access token

  • instance_id – The specific instance to retrieve

  • timeout_s – The underlying request timeout

classmethod get_job(access_token: str, *, job_id: int, timeout_s: int = 4) → DmApiRv

Gets detailed information about a specific Job using the numeric Job record identity

Parameters

  • access_token – A valid DM API access token.

  • job_id – The numeric Job identity

  • timeout_s – The API request timeout

classmethod get_job_by_version(access_token: str, *, job_collection: str, job_job: str, job_version: str, timeout_s: int = 4) → DmApiRv

Gets detailed information about a specific Job using its collection, job and version

Parameters

  • access_token – A valid DM API access token.

  • job_collection – The Job collection, e.g. im-test

  • job_job – The Job, e.g. nop

  • job_version – The Job version, e.g. 1.0.0

  • timeout_s – The API request timeout

classmethod get_job_exchange_rates(access_token: str, *, only_undefined: bool = False, timeout_s: int = 4) → DmApiRv

Gets exchange rates for Jobs.

Parameters

  • access_token – A valid DM API access token

  • only_undefined – True to only include jobs that have no exchange rate

  • timeout_s – The underlying request timeout

classmethod get_project(access_token: str, *, project_id: str, timeout_s: int = 4) → DmApiRv

Gets detailed information about a specific project.

Parameters

  • access_token – A valid DM API access token

  • project_id – The specific project to retrieve

  • timeout_s – The underlying request timeout

classmethod get_project_instances(access_token: str, *, project_id: str, timeout_s: int = 4) → DmApiRv

Gets information about all instances available to you.

Parameters

  • access_token – A valid DM API access token

  • project_id – A valid DM project

  • timeout_s – The underlying request timeout

classmethod get_service_errors(access_token: str, *, include_acknowleged: bool = False, timeout_s: int = 4) → DmApiRv

Gets service errors. You need admin rights to use this method.

Parameters

  • access_token – A valid DM API access token

  • timeout_s – The underlying request timeout

Parem include_acknowleged

True to include acknowledged errors

classmethod get_task(access_token: str, *, task_id: str, event_prior_ordinal: int = 0, event_limit: int = 0, timeout_s: int = 4) → DmApiRv

Gets information about a specific Task

Parameters

  • access_token – A valid DM API access token

  • task_id – The task

  • event_prior_ordinal – The event prior ordinal, Use 0 for the first

  • event_limit – The number of events to return. Use 0 for the default, which depends on the environment, and is typically 500

  • timeout_s – The underlying request timeout

classmethod get_unmanaged_project_file(access_token: str, *, project_id: str, project_file: str, local_file: str, project_path: str = '/', timeout_s: int = 12) → DmApiRv

Get a single unmanaged file from a project path, save it to the filename defined in local_file.

Parameters

  • access_token – A valid DM API access token

  • project_id – The project where the files are present

  • project_file – The name of the file to get

  • local_file – The name to use to write the file to on the client

  • project_path – The path in the project to search for files. The path is relative to the project root and must begin /

  • timeout_s – The underlying request timeout

classmethod get_unmanaged_project_file_with_token(*, token: str, project_id: str, project_file: str, local_file: str, project_path: str = '/', timeout_s: int = 12) → DmApiRv

Like get_unmanaged_project_file(), this method gets a single unmanaged file from a project path. The method uses an Instance-generated callback token rather than a user-access token.

This method is particularly useful in callback routines where a user access token may not be available. Callback tokens expire and can be deleted, and so this function should only be used when a user access token is not available.

Parameters

  • token – A DM-generated token, optionally generated when launching instances in the project

  • project_id – The project where the files are present

  • project_file – The name of the file to get

  • local_file – The name to use to write the file to on the client

  • project_path – The path in the project to search for files. The path is relative to the project root and must begin /

  • timeout_s – The underlying request timeout

classmethod get_version(access_token: Optional[str] = None, *, timeout_s: int = 4) → DmApiRv

Returns the DM-API service version.

Parameters

  • access_token – An optional valid DM API access token (deprecated)

  • timeout_s – The underlying request timeout

classmethod list_project_files(access_token: str, *, project_id: str, project_path: str = '/', include_hidden: bool = False, timeout_s: int = 12) → DmApiRv

Gets a list of project files on a path.

Parameters

  • access_token – A valid DM API access token

  • project_id – The project where the files are present

  • project_path – The path in the project to search for files. The path is relative to the project root and must begin /

  • include_hidden – Include hidden files in the reponse

  • timeout_s – The underlying request timeout

classmethod ping(access_token: str, *, timeout_s: int = 4) → DmApiRv

A handy API method that calls the DM API to ensure the server is responding.

Parameters

  • access_token – A valid DM API access token

  • timeout_s – The underlying request timeout

classmethod put_unmanaged_project_files(access_token: str, *, project_id: str, project_files: Union[str, List[str]], project_path: str = '/', force: bool = False, timeout_per_file_s: int = 120) → DmApiRv

Puts a file, or list of files, into a DM Project using an optional path.

Parameters

  • access_token – A valid DM API access token

  • project_id – The project where the files are to be written

  • project_files – A file or list of files. Leading paths are stripped so the two file files ['dir/file-a.txt', 'file-b.txt'] would be written to the same project directory, i.e. appearing as /file-a.txt and /file-b.txt in the project

  • project_path – The path in the project to write the files. The path is relative to the project root and must begin /

  • force – Files are not written to the project if a file of the same name exists. Here force can be used to over-write files. Files on the server that are immutable cannot be over-written, and doing so will result in an error

  • timeout_per_file_s – The underlying request timeout

classmethod set_admin_state(access_token: str, *, admin: bool, impersonate: Optional[str] = None, timeout_s: int = 4) → DmApiRv

Adds or removes the become-admin state of your account. Only users whose accounts offer administrative capabilities can use this method.

Parameters

  • access_token – A valid DM API access token.

  • admin – True to set admin state

  • impersonate – An optional username to switch to

  • timeout_s – The API request timeout

classmethod set_api_url(url: str, *, verify_ssl_cert: bool = True) → None

Replaces the API URL value, which is otherwise set using the SQUONK2_DMAPI_URL environment variable.

Parameters

  • url – The API endpoint, typically https://example.com/data-manager-api

  • verify_ssl_cert – Use False to avoid SSL verification in request calls

classmethod set_job_exchange_rates(access_token: str, *, rates: Union[Dict[str, str], List[Dict[str, str]]], timeout_s: int = 4) → DmApiRv

Sets exchange rates for Jobs, given one rate or a list of rates.

A rate is a dictionary with keys ‘collection’, ‘job’, ‘version’, and ‘rate’. An optional ‘comment’ can also be provided. The rate is expected to be a string representation of a decimal, i.e. ‘0.05’.

Parameters

  • access_token – A valid DM API access token

  • rates – A rate or a list of rates

  • timeout_s – The underlying request timeout

classmethod start_job_instance(access_token: str, *, project_id: str, name: str, specification: Dict[str, Any], callback_url: Optional[str] = None, callback_context: Optional[str] = None, generate_callback_token: bool = False, debug: Optional[str] = None, timeout_s: int = 4) → DmApiRv

Instantiates a Job Instance in a Project.

Parameters

  • access_token – A valid DM API access token

  • project_id – The project where the files are present

  • name – A name to associate with the Job

  • specification – The Job specification, it must contain keys that define the Job’s collection, job name and version. Job-specific variables are passed in using a variables map in the specification

  • callback_url – An optional URL capable of handling Job callbacks. Must be set if generate_callback_token is used

  • callback_context – An optional context string passed to the callback URL

  • generate_callback_token – True to instruct the DM to generate a token that can be used with some methods instead of a user access token

  • debug – Used to prevent the automatic removal of the Job instance. Only use this if you need to

  • timeout_s – The underlying request timeout

class squonk2.dm_api.DmApiRv(success, msg)

The return value from most of the the DmApi class public methods.

Parameters

  • success – True if the call was successful, False otherwise.

  • msg – API request response content

msg

Alias for field number 1

success

Alias for field number 0

squonk2.dm_api.TEST_PRODUCT_ID: str = 'product-11111111-1111-1111-1111-111111111111'

A test Account Server (AS) Product ID. This ID does not actually exist in the AS but is accepted as valid by the Data Manager for Administrative users and used for testing purposes. It allows the creation of Projects without the need of an AS Product.