The Data Manager API
A module providing simplified assess to the Squonk2 Data Manager API.
The API URL
The URL to the Data Manager API is taken from the environment variable
SQUONK2_DMAPI_URL
if it exists. If you do not want to verify the
SSL certificate you can also set SQUONK2_DMAPI_VERIFY_SSL_CERT
to
anything other than "yes"
, its default value.
If you haven’t set the API URL variables you need to set the Data Manager API URL before you can use any API method.
url = 'https://example.com/data-manager-api'
DmApi.set_api_url(url)
If the Data Manager API is not secure (e.g. you’re developing locally) you can disable the automatic SSL authentication when you set the URL.
dm_api.set_api_url(url, verify_ssl_cert=False)
The API
Python utilities to simplify calls to some parts of the Data Manager API that interact with Projects, Instances (Jobs) and Files.
Note
The URL to the DM API is automatically picked up from the environment variable
SQUONK2_DMAPI_URL
, expected to be of the form https://example.com/data-manager-api.
If the variable isn’t set the user must set it programmatically
using DmApi.set_api_url()
.
- class squonk2.dm_api.DmApi
The DmAPI class provides high-level, simplified access to the DM REST API. You can use the request module directly for finer control. This module provides a wrapper around the handling of the request, returning a simplified namedtuple response value
DmApiRv
- classmethod add_project_editor(access_token: str, *, project_id: str, editor: str, timeout_s: int = 4) DmApiRv
Adds a user to a Project as an Editor.
- Parameters
access_token – A valid DM API access token.
project_id – The Project UUID.
editor – The username to add.
timeout_s – The API request timeout
- classmethod add_project_observer(access_token: str, *, project_id: str, observer: str, timeout_s: int = 4) DmApiRv
Adds a user to a Project as an Observer.
- Parameters
access_token – A valid DM API access token.
project_id – The Project UUID.
editor – The username to add.
timeout_s – The API request timeout
- classmethod create_project(access_token: str, *, project_name: str, as_tier_product_id: str, private: bool = False, timeout_s: int = 4) DmApiRv
Creates a Project, which requires a name and a Product ID (a Data Manger Project Tier Product) obtained from the Account Server.
- Parameters
access_token – A valid DM API access token.
project_name – A unique name.
as_tier_product_id – If no account server is attached any suitable value can be used. If you are an admin user you can also use the reserved value of
product-11111111-1111-1111-1111-111111111111
which is automatically accepted.timeout_s – The API request timeout
- classmethod delete_instance(access_token: str, *, instance_id: str, timeout_s: int = 4) DmApiRv
Deletes an Instance (Application or Job).
When instances are deleted the container is removed along with the instance-specific directory that is automatically created in the root of the project. Any files in the instance-specific directory will be removed.
- Parameters
access_token – A valid DM API access token
instance_id – The instance to delete
timeout_s – The underlying request timeout
- classmethod delete_instance_token(*, instance_id: str, token: str, timeout_s: int = 4) DmApiRv
Deletes a DM API Instance callback token. This API method is not authenticated and therefore does not need an access token. Once the token is deleted no further calls to
DmApi.get_unmanaged_project_file_with_token()
will be possible. Once deleted the token cannot be re-instantiated.- Parameters
instance_id – A valid DM API instance
token – The callback Token associated with the instance
timeout_s – The API request timeout
- classmethod delete_project(access_token: str, *, project_id: str, timeout_s: int = 4) DmApiRv
Deletes a project.
- Parameters
access_token – A valid DM API access token
project_id – The DM-API project id to delete
timeout_s – The underlying request timeout
- classmethod delete_unmanaged_project_files(access_token: str, *, project_id: str, project_files: Union[str, List[str]], project_path: str = '/', timeout_s: int = 4) DmApiRv
Deletes an unmanaged project file, or list of files, on a project path.
- Parameters
access_token – A valid DM API access token
project_id – The project where the files are present
project_files – A file or list of files. Leading paths are stripped so the two file files
['dir/file-a.txt', 'file-b.txt']
would be expected to be in the same project directory, i.e. appearing as/file-a.txt
and/file-b.txt
in the projectproject_path – The path in the project where the files are located. The path is relative to the project root and must begin
/
timeout_s – The underlying request timeout
- classmethod dry_run_job_instance(access_token: str, *, project_id: str, name: str, specification: Dict[str, Any], callback_url: Optional[str] = None, callback_token: Optional[str] = None, callback_context: Optional[str] = None, generate_callback_token: bool = False, debug: Optional[str] = None, timeout_s: int = 4) DmApiRv
Optionally used prior to starting a Job instance, this method checks that Job Instance can be started in a Project, returning the Job command.
- Parameters
access_token – A valid DM API access token
project_id – The project where the files are present
name – A name to associate with the Job
specification – The Job specification, it must contain keys that define the Job’s
collection
,job name
andversion
. Job-specific variables are passed in using avariables
map in the specificationcallback_url – An optional URL capable of handling Job callbacks. Must be set if
generate_callback_token
is usedcallback_token – An optional callback token as an alternative to using
generate_callback_token
callback_context – An optional context string passed to the callback URL
generate_callback_token – True to instruct the DM to generate a token that can be used with some methods instead of a user access token
debug – Used to prevent the automatic removal of the Job instance. Only use this if you need to
timeout_s – The underlying request timeout
- classmethod get_account_server_namespace(access_token: str, *, timeout_s: int = 4) DmApiRv
Gets Datasets available to the caller.
- Parameters
access_token – A valid DM API access token
timeout_s – The underlying request timeout
- classmethod get_account_server_registration(access_token: str, *, timeout_s: int = 4) DmApiRv
Gets Datasets available to the caller.
- Parameters
access_token – A valid DM API access token
timeout_s – The underlying request timeout
- classmethod get_api_url() Tuple[str, bool]
Return the API URL and whether validating the SSL layer.
- classmethod get_available_datasets(access_token: str, *, timeout_s: int = 4) DmApiRv
Gets Datasets available to the caller.
- Parameters
access_token – A valid DM API access token
timeout_s – The underlying request timeout
- classmethod get_available_instances(access_token: str, *, timeout_s: int = 4) DmApiRv
Gets information about all instances available to you.
- Parameters
access_token – A valid DM API access token
timeout_s – The underlying request timeout
- classmethod get_available_jobs(access_token: str, *, timeout_s: int = 4) DmApiRv
Gets a summary list of available Jobs.
- Parameters
access_token – A valid DM API access token.
timeout_s – The API request timeout
- classmethod get_available_projects(access_token: str, *, project_name: Optional[str] = None, timeout_s: int = 4) DmApiRv
Gets information about all projects available to you.
- Parameters
access_token – A valid DM API access token
project_name – An optional project name to use as a filter
timeout_s – The underlying request timeout
- classmethod get_available_tasks(access_token: str, *, exclude_done: bool = False, exclude_purpose: Optional[str] = None, project_id: Optional[str] = None, timeout_s: int = 4) DmApiRv
Gets information about all tasks available to you.
- Parameters
access_token – A valid DM API access token
exclude_done – Set if you want to omit tasks that are ‘done’
exclude_purpose – A comma-separated list of purposes to exclude. Any supported, e.g. DATASET, FILE, INSTANCE, PROJECT
project_id – An optional project ID to limit tasks to
timeout_s – The underlying request timeout
- classmethod get_instance(access_token: str, *, instance_id: str, timeout_s: int = 4) DmApiRv
Gets information about an instance (Application or Job).
- Parameters
access_token – A valid DM API access token
instance_id – The specific instance to retrieve
timeout_s – The underlying request timeout
- classmethod get_job(access_token: str, *, job_id: int, project_id: Optional[str] = None, timeout_s: int = 4) DmApiRv
Gets detailed information about a specific Job using the numeric Job record identity.
- Parameters
access_token – A valid DM API access token.
job_id – The numeric Job identity
project_id – An optional Project the Job is to be run in, e.g.
project-0000
timeout_s – The API request timeout
- classmethod get_job_by_version(access_token: str, *, job_collection: str, job_job: str, job_version: str, project_id: str = '', timeout_s: int = 4) DmApiRv
Gets detailed information about a specific Job using its
collection
,job
andversion
, using an optional targetproject_id
.- Parameters
access_token – A valid DM API access token.
job_collection – The Job collection, e.g.
im-test
job_job – The Job, e.g.
nop
job_version – The Job version, e.g.
1.0.0
project_id – An optional Project the Job is to be run in, e.g.
project-0000
timeout_s – The API request timeout
- classmethod get_job_exchange_rates(access_token: str, *, only_undefined: bool = False, timeout_s: int = 4) DmApiRv
Gets exchange rates for Jobs.
- Parameters
access_token – A valid DM API access token
only_undefined – True to only include jobs that have no exchange rate
timeout_s – The underlying request timeout
- classmethod get_jobs(access_token: str, *, project_id: Optional[str] = None, timeout_s: int = 4) DmApiRv
Gets summary information about all Jobs.
- Parameters
access_token – A valid DM API access token.
project_id – An optional Project the Job is to be run in, e.g.
project-0000
timeout_s – The API request timeout
- classmethod get_project(access_token: str, *, project_id: str, timeout_s: int = 4) DmApiRv
Gets detailed information about a specific project.
- Parameters
access_token – A valid DM API access token
project_id – The specific project to retrieve
timeout_s – The underlying request timeout
- classmethod get_project_instances(access_token: str, *, project_id: str, timeout_s: int = 4) DmApiRv
Gets information about all instances available to you.
- Parameters
access_token – A valid DM API access token
project_id – A valid DM project
timeout_s – The underlying request timeout
- classmethod get_service_errors(access_token: str, *, include_acknowledged: bool = False, timeout_s: int = 4) DmApiRv
Gets service errors. You need admin rights to use this method.
- Parameters
access_token – A valid DM API access token
include_acknowledged – True to include acknowledged errors
timeout_s – The underlying request timeout
- classmethod get_task(access_token: str, *, task_id: str, event_prior_ordinal: int = 0, event_limit: int = 0, timeout_s: int = 4) DmApiRv
Gets information about a specific Task
- Parameters
access_token – A valid DM API access token
task_id – The task
event_prior_ordinal – The event prior ordinal, Use 0 for the first
event_limit – The number of events to return. Use 0 for the default, which depends on the environment, and is typically 500
timeout_s – The underlying request timeout
- classmethod get_tasks(access_token: str, *, exclude_done: bool = False, exclude_removal: bool = False, exclude_purpose: Optional[str] = None, project_id: Optional[str] = None, instance_callback_context: Optional[str] = None, timeout_s: int = 4) DmApiRv
Gets information about a range of Tasks
- Parameters
access_token – A valid DM API access token
exclude_done – Set if you do not want to see completed Tasks
exclude_removal – Set if you do not want to see removal Tasks
exclude_purpose – A dot-separated string of purposes to exclude. From INSTANCE, FILE or DATASET
project_id – Limit tasks to the given Project
instance_callback_context – Limit tasks to those for Instances with the given callback context
timeout_s – The underlying request timeout
- classmethod get_unmanaged_project_file(access_token: str, *, project_id: str, project_file: str, local_file: str, project_path: str = '/', timeout_s: int = 12) DmApiRv
Get a single unmanaged file from a project path, save it to the filename defined in local_file.
- Parameters
access_token – A valid DM API access token
project_id – The project where the files are present
project_file – The name of the file to get
local_file – The name to use to write the file to on the client
project_path – The path in the project to search for files. The path is relative to the project root and must begin
/
timeout_s – The underlying request timeout
- classmethod get_unmanaged_project_file_with_token(*, token: str, project_id: str, project_file: str, local_file: str, project_path: str = '/', timeout_s: int = 12) DmApiRv
Like
get_unmanaged_project_file()
, this method gets a single unmanaged file from a project path. The method uses an Instance-generated callback token rather than a user-access token.This method is particularly useful in callback routines where a user access token may not be available. Callback tokens expire and can be deleted, and so this function should only be used when a user access token is not available.
- Parameters
token – A DM-generated token, optionally generated when launching instances in the project
project_id – The project where the files are present
project_file – The name of the file to get
local_file – The name to use to write the file to on the client
project_path – The path in the project to search for files. The path is relative to the project root and must begin
/
timeout_s – The underlying request timeout
- classmethod get_version(access_token: Optional[str] = None, *, timeout_s: int = 4) DmApiRv
Returns the DM-API service version.
- Parameters
access_token – An optional valid DM API access token (deprecated)
timeout_s – The underlying request timeout
- classmethod list_project_files(access_token: str, *, project_id: str, project_path: str = '/', include_hidden: bool = False, timeout_s: int = 12) DmApiRv
Gets a list of project files on a path.
- Parameters
access_token – A valid DM API access token
project_id – The project where the files are present
project_path – The path in the project to search for files. The path is relative to the project root and must begin
/
include_hidden – Include hidden files in the reponse
timeout_s – The underlying request timeout
- classmethod ping(access_token: str, *, timeout_s: int = 4) DmApiRv
A handy API method that calls the DM API to ensure the server is responding.
- Parameters
access_token – A valid DM API access token
timeout_s – The underlying request timeout
- classmethod put_job_manifest(access_token: str, *, url: str, header: Optional[str] = None, params: Optional[str] = None, timeout_s: int = 4) DmApiRv
Puts a Job Manifest onto server. The action requires the token to be that of an admin user.
- Parameters
access_token – A valid DM API access token
url – The location of the Manifest (typically a GitHub repository URL)
header – An optional JSON string of header keys and values
params – An optional JSON string of parameter keys and values
timeout_s – The underlying request timeout
- classmethod put_unmanaged_project_files(access_token: str, *, project_id: str, project_files: Union[str, List[str]], project_path: str = '/', force: bool = False, timeout_per_file_s: int = 120) DmApiRv
Puts a file, or list of files, into a DM Project using an optional path.
- Parameters
access_token – A valid DM API access token
project_id – The project where the files are to be written
project_files – A file or list of files. Leading paths are stripped so the two file files
['dir/file-a.txt', 'file-b.txt']
would be written to the same project directory, i.e. appearing as/file-a.txt
and/file-b.txt
in the projectproject_path – The path in the project to write the files. The path is relative to the project root and must begin
/
force – Files are not written to the project if a file of the same name exists. Here
force
can be used to over-write files. Files on the server that are immutable cannot be over-written, and doing so will result in an errortimeout_per_file_s – The underlying request timeout
- classmethod remove_project_editor(access_token: str, *, project_id: str, editor: str, timeout_s: int = 4) DmApiRv
Removes a user as an Editor from a Project.
- Parameters
access_token – A valid DM API access token.
project_id – The Project UUID.
editor – The username to remove.
timeout_s – The API request timeout
- classmethod remove_project_observer(access_token: str, *, project_id: str, observer: str, timeout_s: int = 4) DmApiRv
Removes a user as an Observer from a Project.
- Parameters
access_token – A valid DM API access token.
project_id – The Project UUID.
observer – The username to remove.
timeout_s – The API request timeout
- classmethod set_admin_state(access_token: str, *, admin: bool, impersonate: Optional[str] = None, timeout_s: int = 4) DmApiRv
Adds or removes the
become-admin
state of your account. Only users whose accounts offer administrative capabilities can use this method.- Parameters
access_token – A valid DM API access token.
admin – True to set admin state
impersonate – An optional username to switch to
timeout_s – The API request timeout
- classmethod set_api_url(url: str, *, verify_ssl_cert: bool = True) None
Replaces the API URL value, which is otherwise set using the
SQUONK2_DMAPI_URL
environment variable.- Parameters
url – The API endpoint, typically https://example.com/data-manager-api
verify_ssl_cert – Use False to avoid SSL verification in request calls
- classmethod set_job_exchange_rates(access_token: str, *, rates: Union[Dict[str, str], List[Dict[str, str]]], timeout_s: int = 4) DmApiRv
Sets exchange rates for Jobs, given one rate or a list of rates.
A rate is a dictionary with keys ‘collection’, ‘job’, ‘version’, and ‘rate’. An optional ‘comment’ can also be provided. The rate is expected to be a string representation of a decimal, i.e. ‘0.05’.
- Parameters
access_token – A valid DM API access token
rates – A rate or a list of rates
timeout_s – The underlying request timeout
- classmethod start_job_instance(access_token: str, *, project_id: str, name: str, specification: Dict[str, Any], callback_url: Optional[str] = None, callback_token: Optional[str] = None, callback_context: Optional[str] = None, generate_callback_token: bool = False, debug: Optional[str] = None, timeout_s: int = 4) DmApiRv
Instantiates a Job Instance in a Project.
- Parameters
access_token – A valid DM API access token
project_id – The project where the files are present
name – A name to associate with the Job
specification – The Job specification, it must contain keys that define the Job’s
collection
,job name
andversion
. Job-specific variables are passed in using avariables
map in the specificationcallback_url – An optional URL capable of handling Job callbacks. Must be set if
generate_callback_token
is usedcallback_token – An optional callback token as an alternative to using
generate_callback_token
callback_context – An optional context string passed to the callback URL
generate_callback_token – True to instruct the DM to generate a token that can be used with some methods instead of a user access token
debug – Used to prevent the automatic removal of the Job instance. Only use this if you need to
timeout_s – The underlying request timeout
- class squonk2.dm_api.DmApiRv(success: bool, msg: Dict[Any, Any])
The return value from most of the the DmApi class public methods.
- Parameters
success – True if the call was successful, False otherwise.
msg – API request response content
- squonk2.dm_api.TEST_PRODUCT_ID: str = 'product-11111111-1111-1111-1111-111111111111'
A test Account Server (AS) Product ID. This ID does not actually exist in the AS but is accepted as valid by the Data Manager for Administrative users and used for testing purposes. It allows the creation of Projects without the need of an AS Product.