Command Line Tools

Example Use Cases

Download public files for a given project/user

You can use ohpub-download on the command line to download all public files associated with a project. Here is an example that will download all public 23andMe files:

mkdir 23andme # create new folder to store the data in
ohpub-download --source direct-sharing-128 --directory 23andme

To do the same for all public files of a given user:

mkdir gedankenstuecke # create new folder to store the data in
ohpub-download --username gedankenstuecke --directory gedankenstuecke

Download private files from members of your project

If you are running a project yourself you can use a master_access_token you can get from the Open Humans website to download the files shared with you.

Here is an example of how to do this:

mkdir my_downloaded_data # create new folder to store the data in
ohproj-download --master-token my_master_access_token --directory my_downloaded_data

Upload files to the accounts of your project members

Uploading data into your members accounts is two-step procedure using both ohproj-upload-metadata as well as ohproj-upload, as you first need to draft some metadata that will go along with the files.

Both commands expect a directory that contains sub-directories named after your members project_member_id. An example directory structure with files could look like this:

  • member_data/
    • 01234567/
      • testdata.json
      • testdata.txt
    • 12345678/
      • testdata.json
      • testdata.txt

To upload this data we need to have a CSV file that contains the metadata. We can draft one using the following command:

ohproj-upload-metadata -d member_data --create-csv member_data_metadata.csv

The resulting CSV will look like this:

$ cat member_data_metadata.csv =>
project_member_id,filename,tags,description,md5,creation_date
01234567,testdata.txt,,,fa61a92e21a2597900cbde09d8ddbc1a,2016-08-23T15:23:22.277060+00:00
01234567,testdata.json,json,,577da9879649acaf17226a6461bd19c8,2016-08-23T16:06:16.415039+00:00
12345678,testdata.txt,,,fa61a92e21a2597900cbde09d8ddbc1a,2016-09-20T10:10:59.863201+00:00
12345678,testdata.json,json,,577da9879649acaf17226a6461bd19c8,2016-09-20T10:10:59.859201+00:00

Edit this CSV with a text or spreadsheet editor of your choice to contain the metadata and then save it as a CSV again:

$ cat member_data_metadata.csv =>
project_member_id,filename,tags,description,md5,creation_date
1234567,testdata.txt,"txt, verbose-data",Complete test data in text format.,fa61a92e21a2597900cbde09d8ddbc1a,2016-08-23T15:23:22.277060+00:00
1234567,testdata.json,"json, metadata",Summary metadata in JSON format.,577da9879649acaf17226a6461bd19c8,2016-08-23T16:06:16.415039+00:00
12345678,testdata.txt,"txt, verbose-data",Complete test data in text format.,fa61a92e21a2597900cbde09d8ddbc1a,2016-09-20T10:10:59.863201+00:00
12345678,testdata.json,"json, metadata",Summary test data JSON.,577da9879649acaf17226a6461bd19c8,2016-09-20T10:10:59.859201+00:00

We filled in the tags as a "-escaped and comma-separated list as well as a text-description. With this we can now perform the actual upload like this:

ohproj-upload -T YOUR_MASTER_ACCESS_TOKEN --metadata-csv member_data_metadata.csv -d member_data

This will upload the data from the member_data directory along with the metadata specified in member_data_metadata.csv.

Using an OAuth2 access_token

The CLI tools also accept the usage of an OAuth2 access_token``s instead of an ``master_access_token. This enables you to use the CLI tools in your own OAuth2 applications if you’re not using Python for your development.

Here is an example to download all the data shared by a single member through the command line:

ohproj-download -t personal_access_token -m '12345678' --directory download_directory

List of commands and their parameters

ohpub-download

Command line tools for downloading public data.

ohpub-download [OPTIONS]

Options

-s, --source <source>

the source to download files from

-u, --username <username>

the user to download files from

-d, --directory <directory>

the directory for downloaded files

-m, --max-size <max_size>

the maximum file size to download

-q, --quiet

Report ERROR level logging to stdout

--debug

Report DEBUG level logging to stdout.

ohproj-download

Command line function for downloading data from project members to the target directory. For more information visit download.

ohproj-download [OPTIONS]

Options

-d, --directory <directory>

Target directory for downloaded files. [required]

-T, --master-token <master_token>

Project master access token.

-m, --member <member>

Project member ID.

-t, --access-token <access_token>

OAuth2 user access token.

-s, --source <source>

Only download files from this source.

--project-data

Download this project’s own data.

--max-size <max_size>

Maximum file size to download. [default: 128m]

-v, --verbose

Report INFO level logging to stdout

--debug

Report DEBUG level logging to stdout.

--memberlist <memberlist>

Text file with whitelist IDs to retrieve

--excludelist <excludelist>

Text file with blacklist IDs to avoid

--id-filename

Prepend filenames with IDs to ensure uniqueness.

ohproj-download-metadata

Command line function for downloading metadata. For more information visit download_metadata.

ohproj-download-metadata [OPTIONS]

Options

-T, --master-token <master_token>

Project master access token. [required]

-v, --verbose

Show INFO level logging

--debug

Show DEBUG level logging.

--output-csv <output_csv>

Output project metedata CSV [required]

ohproj-upload-metadata

Command line function for drafting or reviewing metadata files. For more information visit upload_metadata.

ohproj-upload-metadata [OPTIONS]

Options

-d, --directory <directory>

Target directory [required]

--create-csv <create_csv>

Create draft CSV metadata

--review <review>

Review existing metadata file

--max-size <max_size>

Maximum file size to consider. [default: 128m]

-v, --verbose

Show INFO level logging

--debug

Show DEBUG level logging.

ohproj-upload

Command line function for uploading files to OH. For more information visit upload.

ohproj-upload [OPTIONS]

Options

-d, --directory <directory>

Target directory for downloaded files. [required]

--metadata-csv <metadata_csv>

CSV file containing file metadata. [required]

-T, --master-token <master_token>

Project master access token.

-m, --member <member>

Project member ID.

-t, --access-token <access_token>

OAuth2 user access token.

--safe

Do not overwrite files in Open Humans.

--sync

Delete files not present in local directories.

--max-size <max_size>

Maximum file size to download. [default: 128m]

-v, --verbose

Report INFO level logging to stdout

--debug

Report DEBUG level logging to stdout.

ohproj-oauth2-token-exchange

Command line function for obtaining the refresh token/code. For more information visit oauth2_token_exchange.

ohproj-oauth2-token-exchange [OPTIONS]

Options

-cid, --client_id <client_id>

client id of user. [required]

-cs, --client_secret <client_secret>

client secret of user. [required]

-re_uri, --redirect_uri <redirect_uri>

redirect_uri of user [required]

--base_url <base_url>

base url of Open Humans [default: https://www.openhumans.org/]

--code <code>

code of user

-rt, --refresh_token <refresh_token>

refresh token of user

ohproj-oauth2-url

Command line function for obtaining the Oauth2 url. For more information visit oauth2_auth_url.

ohproj-oauth2-url [OPTIONS]

Options

-r, --redirect_uri <redirect_uri>

Redirect URL for project

-c, --client_id <client_id>

Client ID for project [required]

--base_url <base_url>

Base URL

ohproj-message

Command line function for sending email to a single user or in bulk. For more information visit message.

ohproj-message [OPTIONS]

Options

-s, --subject <subject>

subject [required]

-m, --message_body <message_body>

compose message [required]

-at, --access_token <access_token>

access token [required]

--all_members <all_members>

all members [default: False]

--project_member_ids <project_member_ids>

list of comma-separated project_member_ids. Example argument: “ID1, ID2”

-v, --verbose

Show INFO level logging

--debug

Show DEBUG level logging.

ohproj-delete

Command line function for deleting files. For more information visit delete_file.

ohproj-delete [OPTIONS]

Options

-T, --access_token <access_token>

Access token [required]

-m, --project_member_id <project_member_id>

Project Member ID [required]

-b, --file_basename <file_basename>

File basename

-i, --file_id <file_id>

File ID

--all_files <all_files>

Delete all files [default: False]

ohapi.command_line.download(directory, master_token=None, member=None, access_token=None, source=None, project_data=False, max_size='128m', verbose=False, debug=False, memberlist=None, excludelist=None, id_filename=False)

Download data from project members to the target directory.

Unless this is a member-specific download, directories will be created for each project member ID. Also, unless a source is specified, all shared sources are downloaded and data is sorted into subdirectories according to source.

Projects can optionally return data to Open Humans member accounts. If project_data is True (or the “–project-data” flag is used), this data (the project’s own data files, instead of data from other sources) will be downloaded for each member.

Parameters:
  • directory – This field is the target directory to download data.
  • master_token – This field is the master access token for the project. It’s default value is None.
  • member – This field is specific member whose project data is downloaded. It’s default value is None.
  • access_token – This field is the user specific access token. It’s default value is None.
  • source – This field is the data source. It’s default value is None.
  • project_data – This field is data related to particular project. It’s default value is False.
  • max_size – This field is the maximum file size. It’s default value is 128m.
  • verbose – This boolean field is the logging level. It’s default value is False.
  • debug – This boolean field is the logging level. It’s default value is False.
  • memberlist – This field is list of members whose data will be downloaded. It’s default value is None.
  • excludelist – This field is list of members whose data will be skipped. It’s default value is None.
ohapi.command_line.download_metadata(master_token, output_csv, verbose=False, debug=False)

Output CSV with metadata for a project’s downloadable files in Open Humans.

Parameters:
  • master_token – This field is the master access token for the project.
  • output_csv – This field is the target csv file to which metadata is written.
  • verbose – This boolean field is the logging level. It’s default value is False.
  • debug – This boolean field is the logging level. It’s default value is False.
ohapi.command_line.set_log_level(debug, verbose)

Function for setting the logging level.

Parameters:
  • debug – This boolean field is the logging level.
  • verbose – This boolean field is the logging level.
ohapi.command_line.upload(directory, metadata_csv, master_token=None, member=None, access_token=None, safe=False, sync=False, max_size='128m', mode='default', verbose=False, debug=False)

Upload files for the project to Open Humans member accounts.

If using a master access token and not specifying member ID:

(1) Files should be organized in subdirectories according to project member ID, e.g.:

main_directory/01234567/data.json main_directory/12345678/data.json main_directory/23456789/data.json
  1. The metadata CSV should have the following format:

    1st column: Project member ID 2nd column: filenames 3rd & additional columns: Metadata fields (see below)

If uploading for a specific member: (1) The local directory should not contain subdirectories. (2) The metadata CSV should have the following format: 1st column: filenames 2nd & additional columns: Metadata fields (see below)

The default behavior is to overwrite files with matching filenames on Open Humans, but not otherwise delete files. (Use –safe or –sync to change this behavior.)

If included, the following metadata columns should be correctly formatted: ‘tags’: should be comma-separated strings ‘md5’: should match the file’s md5 hexdigest ‘creation_date’, ‘start_date’, ‘end_date’: ISO 8601 dates or datetimes

Other metedata fields (e.g. ‘description’) can be arbitrary strings. Either specify sync as True or safe as True but not both.

Parameters:
  • directory – This field is the target directory from which data will be uploaded.
  • metadata_csv – This field is the filepath of the metadata csv file.
  • master_token – This field is the master access token for the project. It’s default value is None.
  • member – This field is specific member whose project data is downloaded. It’s default value is None.
  • access_token – This field is the user specific access token. It’s default value is None.
  • safe – This boolean field will overwrite matching filename. It’s default value is False.
  • sync – This boolean field will delete files on Open Humans that are not in the local directory. It’s default value is False.
  • max_size – This field is the maximum file size. It’s default value is None.
  • mode – This field takes three value default, sync, safe. It’s default value is ‘default’.
  • verbose – This boolean field is the logging level. It’s default value is False.
  • debug – This boolean field is the logging level. It’s default value is False.
ohapi.command_line.upload_metadata(directory, create_csv='', review='', max_size='128m', verbose=False, debug=False)

Draft or review metadata files for uploading files to Open Humans. The target directory should either represent files for a single member (no subdirectories), or contain a subdirectory for each project member ID.

Parameters:
  • directory – This field is the directory for which metadata has to be created.
  • create_csv – This field is the output filepath to which csv file will be written.
  • max_size – This field is the maximum file size. It’s default value is None.
  • verbose – This boolean field is the logging level. It’s default value is False.
  • debug – This boolean field is the logging level. It’s default value is False.