Pipeliner CLI

The pipeliner CLI is a python cli program which simplifies iterations with the pipeliner API.

Installation

The cli can be installed using your preferred python package manager.

pippoetry

pip install dp-cli

poetry add dp-cli

Command Reference

init

Description

The init command creates or updates a local pipeliner config file, located at ~/.pipeliner/config. It allows users to set some defaults for other pipeliner cli commands. The defaults set in the pipeliner config file can be overridden by passing the argument manually to the pipeliner cli command.

Synopsis

pipeliner init
[--base-branch]
[--github-org]
[--github-token]
[--api-key]
[--host]

Options

--base-branch (string) Sets a default branch for pipeliner pull requests to be created against.

--github-org (string) Sets a default GitHub Organisation for pipeliner to create pull requests within.

--github-token (string) Sets a default GitHub token for pipeliner to authenticate to GitHub repository.

--api-key (string) Sets default Pipeliner API Key to authenticate requests to pipeliner.

--host (string) Sets the pipeliner host to send requests too, defaults to api.datapipeliner.io

submit

Description

The submit command takes a local mapping specification file and submits it to pipeliner. Pipeliner will generate the ETL code based on your mapping specification and

Synopsis

pipeliner submit
[--file]
[--branch]
[--repo-name]
[--layer-name]
[--system-name]
[--base-branch]
[--catalog-only]
[--github-org]
[--github-token]
[--api-key]
[--host]

Options

--file (string) Path to the mapping specification file on the local machine.

--repo-name (string) Name of the GitHub repository pipeliner should commit the code too.

--branch (string) The name of the branch pipeliner should commit the code to.

--layer-name (string) The name of the target layer in the datalake the mapping specification defines tables for. e.g. silver

--system-name (string) The business name of the system the mapping specification if for. e.g. employees

--base-branch (string) Sets a default branch for pipeliner pull requests to be created against.

--catalog-only (boolean) If true only the infrastructure for the tables definition will be produced. If false, all outputs will be produced. Defaults to false.

--github-org (string) Sets a default GitHub Organisation for pipeliner to create pull requests within.

--github-token (string) Sets a default GitHub token for pipeliner to authenticate to GitHub repository.

--api-key (string) Sets default Pipeliner API Key to authenticate requests to pipeliner.

--host (string) Sets the pipeliner host to send requests too, defaults to api.datapipeliner.io

Note

The --layer-name and --system-name arguments will alter the directory heirachy for code pipelienr commits to the repository. It will also dictate how the data is organised in the data lake.