User Manual

Get PyPortal

PyPortal is the Portal's python package to interact with the data in the CP Portal programatically and to make data changes in a bulk.

With Pyportal, you can:

• Create Annotation Sets
• Add Collection Tags to annotation sets
• Delete Annotation Sets
• Download Annotations
• Download Slides
• Download Raster Annotations
• Upload Annotations
• Upload Landmarks
• Extend Landmarks
• Upload Slides
• Upload Raster Annotations

Download the PyPortal Windows executable

Windows users should download the package from this link: https://gitlab.img.astrazeneca.net/portal/pyportal/-/releases

The downloaded executable can be placed in any place in your operating system. The package should be called within the terminal (NOT by double click).

Image: Use the marked link to download the PyPortal's Windows executable

Running the GxP Script

For raster annotation operations and Mac/Linux support, PyPortal includes a GxP script.

After downloading the latest PyPortal's release, the GxP script can be found at gxp-scripts/pyportal-cli.py.

curl -LsSf https://astral.sh/uv/install.sh | sh

powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

GxP Compliance and Reproducibility:

The GxP script uses locked dependencies to ensure reproducible results. The --locked flag enforces use of the lock file when running commands:

uv run --locked gxp-scripts/pyportal-cli.py [command] [options]

PyPortal Python API and other platforms

• To use PyPortal from other Python applications or a Jupyter notebook, it must be installed as a Python package as described in this document
• This also allows use of PyPortal on other platforms such as Windows, Linux (SCP, Docker) and Mac OS

Running PyPortal's help

Navigate to the folder where you download the pyportal.exe file to.

In the following example, the user Alex has downlaod the pyportal.exe file to the Downloads folder on their machine.

cd C:\Users\Alex\Downloads

They can run pyportal from the folder by typing:

pyportal.exe -h

Alternatively, they can run the package help from any place by calling it using the full path:

C:\Users\Alex\Downloads\pyportal.exe -h

To verify the package version they can run the following command:

C:\Users\Alex\Downloads\pyportal.exe version

Or, if they have navigated to the folder where the pyportal.exe is located, they can use the following command:

pyportal.exe version

Setup PyPortal

To perform any action with the PyPortal, an API key is required. It can be generated in your user settings in the CP Portal by clicking on your user name.

Image: Generate API key page in the CP Portal

Please note that:

• To generate an API key you will be asked to re-authenticate yourself
• Your API key will only be shown right after generation. Please keep it in a secure place
• If you forget your API key, you can generate a new one. The old key becomes invalid
• Please check out the Accessing the Portal via API article if you want to get more information about API key

Setting the API key for PyPortal usage

You can provide API key as a parameter when calling pyportal.exe:

pyportal.exe ds -c [PATH_TO_CONFIG_FILE] --portal-api-key [API_KEY]

However, it is recommended (from security and simplicity reasons) to store it in a .portal_key file in your user home directory, e.g. in C:\Users\Alex\.portal_key.

Please create that file using a text editor, e.g. with Notepad.

To do that, please navigate to your home directory and create new text document:

Image: User home folder where to place the API key file

The name should be exactly .portal_key. Note the dot at the start!

In this file, paste the API key you copied from the CP Portal as its only content. As an example (note that this API key is invalid), this should be the only text present in the key file:

WIFIfKqmOG93ZWeCFkVj2ZtpOCqhKkF2g3d9t672

Turn on showing file extensions, to be sure that file name is exactly .portal_key, and not .portal_key.txt

Image: Show file extensions in Windows explorer

In case of modifying the file name, one can be notified to confirm change of the extension, in such a situation please click yes.

Image: Confirm the change of the file extension in Windows explorer

Running PyPortal on AstraZeneca lab PCs (Cambridge, Gaithersburg)

From some laboratory sites, additional PC settings are required in order to make successful calls with PyPortal.

In some situations proxy needs to be set in order to connect with our infrastructure.

On Windows, simply set the following variables:

set HTTP_PROXY=http://azpzen.astrazeneca.net:9480
set HTTPS_PROXY=http://azpzen.astrazeneca.net:9480

On Linux, set the following variables:

export HTTP_PROXY=http://azpzen.astrazeneca.net:9480
export HTTPS_PROXY=http://azpzen.astrazeneca.net:9480

Further information on PyPortal

For more information please see the information in the package: https://gitlab.img.astrazeneca.net/portal/pyportal#pyportal

Create annotation sets

To create annotation sets, one should use the built-in functionality create-annotation-sets or the shortcut version cas.

To see all available options for the create-annotation-sets command, type:

pyportal.exe cas -h

Annotation Sets can be created by calling the following command:

pyportal.exe cas -p [PROJECT_ID] -s [ANNOTATION_SET_NAME_TO_BE_CREATED]

The Portal project ID can be copied from your browser's address bar.

Image: Project ID as part of the URL in the address bar

As an outcome, messages similar to these should appear in your command line:

pyportal.exe cas -p 619b9c1e30325cc7cc33ec1b --annotation-set test_test_test
PyPortal v2.5.0

Creating annotation sets...

100%|██████████| 1/1 [00:00<00:00,  3.41slide/s]
Successfully created 10 annotation sets with name 'test_test_test'

With warnings:
Slide ID '61a4a20957db16fb8cfe3a25' in project '619b9c1e30325cc7cc33ec1b' already has annotation set 'test_test_test'. No annotation set was created for this slide.
Slide ID '61a4a46a57db16a8bdfe3acd' in project '619b9c1e30325cc7cc33ec1b' already has annotation set 'test_test_test'. No annotation set was created for this slide.

Create annotation sets for specific slides

To be more specific when creating annotation sets, you can provide --slide-ids argument in the command line. This will create annotation sets only for the specified slide(s).

If an annotation set with such a name already exists, warnings will be printed, and an annotation set will not be created. It will not be overwritten, nor any annotations are removed.

pyportal.exe cas -p [PROJECT_ID] -s [ANNOTATION_SET_NAME_TO_BE_CREATED] --slide-ids "[SLIDE_ID_1]" "[SLIDE_ID_2]" "[...]"

As an outcome, messages similar to these should appear in your command line:

pyportal.exe cas -p 619b9c1e30325cc7cc33ec1b --annotation-set test_test_test --slide-ids

PyPortal v2.5.0

Creating annotation sets...

100%|██████████| 1/1 [00:00<00:00,  3.41slide/s]
Successfully created 1 annotation sets with name 'test_test_test'

With warnings:
Slide ID '61a4a20957db16fb8cfe3a25' in project '619b9c1e30325cc7cc33ec1b' already has annotation set 'test_test_test'. No annotation set was created for this slide.

With errors:
SlideID '61d57795309f68ced5693402' not found in project with ID '619b9c1e30325cc7cc33ec1b'

Usage of config file instead of CLI arguments

As in other commands, here one can also use a config file to have a simpler call in the command line. The following config file: create_annotation_sets_config.yml contains the same parameters as in the above call.

create_annotation_sets_config.yml

project_id: 619b9c1e30325cc7cc33ec1b
annotation_set: "test_test_test"
slide_ids:
  - 61a4a20957db16fb8cfe3a25
  - 61d57795309f68ced5693402
  - 61a4a46a57db16a8bdfe3acd

And the usage is as follows:

pyportal.exe cas --config create_annotation_sets_config.yml

PyPortal v2.5.0

Creating annotation sets...

100%|██████████| 1/1 [00:00<00:00,  3.41slide/s]
Successfully created 1 annotation sets with name 'test_test_test'

With warnings:
Slide ID '61a4a20957db16fb8cfe3a25' in project '619b9c1e30325cc7cc33ec1b' already has annotation set 'test_test_test'. No annotation set was created for this slide.

With errors:
SlideID '61d57795309f68ced5693402' not found in project with ID '619b9c1e30325cc7cc33ec1b'

All available options for creating annotation sets

Add collection tags to annotation sets

Collection tags can be used to mark the latest version of annotations in one or several annotations sets. After tagging, the annotations can be downloaded using the tag name. This way it is possible to set a mark on a particular state of the annotations and use this to retrieve the exact same state again and again, regardless of the changes that were done in the meantime.

The creation of collection tags can be done through Pyportal. You can use the command create-vector-annotation-collection or the short-cut version cvac cvac

To see all options for the create-vector-annotation-collection command, type:

pyportal.exe cvac --help

Collection tags can be created calling the following command:

pyportal.exe cvac -p [PROJECT_ID] -t [COLLECTION_TAG_NAME]

The Portal project ID can be copied from your browser's address bar.

Image: Project ID as part of the URL in the address bar

Create collection tags for specific annotation sets

To be more specific when creating collection tags, you can provide --set-names argument in the command line. This will create collection tags only for the specified annotation sets.

pyportal.exe cvac -p [PROJECT_ID] -t [COLLECTION_TAG_NAME] --set-names "[SET_NAME_1]" "[SET_NAME_2]" "[...]"

All available options for adding collection tags to annotation sets

Delete annotation sets

To delete annotations, one should use the built-in functionality delete-annotation-sets or the shortcut version das.

To see all available options for the delete-annotation-sets command, type:

pyportal.exe das -h

Annotation sets can be deleted from the CP Portal by calling the following command:

pyportal.exe das -p [PROJECT_ID] -s [ANNOTATION_SET_NAME]

The Portal project ID can be copied from your browser's address bar.

Image: Project ID as part of the URL in the address bar

Delete annotation sets from specific slides in a project

To delete annotation sets only from specific slides, one can provide --slide-file-names argument in the command line. In which case, annotation sets will be deleted from only the specified slide(s) if they exist there.

pyportal.exe das -p [PROJECT_ID] -s [ANNOTATION_SET_NAME] --slide-file-names "[SLIDE_NAME_1]" "[SLIDE_NAME_2]" "[...]"

Alternatively to providing slide names, you can provide --slide-ids argument in the command line. In which case, annotation sets will be deleted from only the specified slide(s) if they exist there.

pyportal.exe das -p [PROJECT_ID] -s [ANNOTATION_SET_NAME] --slide-ids "[SLIDE_ID_TAKEN_FROM_BROWSER_1]" "[SLIDE_ID_TAKEN_FROM_BROWSER_2]" "[...]"

All available options for deleting annotations

Download annotation sets

To download annotations, one should use the built-in functionality download-annotations or the shortcut version da.

To see all available options for the download-annotations command, type:

pyportal.exe da -h

To download annotations "as is" into the GeoJSON format, two parameters have to be provided:

• Exactly one project ID which you want to download from
• Exactly one output location to store the files

Download all annotations

In general, the download annotations command to download all annotation from a project, looks like this:

pyportal.exe da -p [PROJECT_ID] -o [ANNOTATION_OUTPUT_FOLDER]

The Portal project ID can be copied from your browser's address bar.

Image: Project ID as part of the URL in the address bar

Usage of a config file

There are many options for downloading annotations (see the table of all options for downloading annotations command). Entering many options into the command might be hard to read/edit, so a special config file can be created in the YAML format. It will give a better overview of the selected options and allow for easy reuse for subsequent downloads. The names of the options are the same as for the client interface, however, "-" (minus) signs between the words need to be replaced with "_" (underscores).

Here is an example annotations_config.yaml file:

project_id: 5e57b8d248dbe6f07e5f44a0
output_path: C:\Users\Alex\Downloads\annotations
format: geojson

The format used in the above example indicates the fact that more options can be included in the config file. The format is not required, the default value geojson is used if it's not provided. Please refer to the table with all options to learn more about all available options that can be used as Command-Line parameters or in the config file.

Using the YAML config file with PyPortal:

pyportal.exe da --config annotations_config.yaml

Download sets

To download annotations from a specific annotation set, one can use the flag --annotation-set:

pyportal.exe da --annotation-set [ANNOTATION_SET_NAME] -o [ANNOTATION_OUTPUT_FOLDER] -p [PROJECT_ID]

Or set the following example value in a config file annotations_config_with_set_name.yaml and use it as in other calls:

project_id: 5e57b8d248dbe6f07e5f44a0
output_path: C:\Users\Alex\Downloads\annotations
annotation_set: Validation

pyportal.exe da --config annotations_config_with_set_name.yaml

As a result of this call, annotations will be placed in the folder 'C:\Users\Alex\Downloads\annotations' in the following structure:

C:\Users\Alex\Downloads\annotations/
└── Validation
    ├── slide_name_1.geojson
    ├── slide_name_2.geojson
    ...
    └── slide_name_n.geojson

Download from specific slides

To download only annotations for a specific set of slides, one can use the flag --slide-ids. In that case, the unique ID of a slide, NOT the file name needs to be provided. You can specify more than one ID.

pyportal.exe da --slide-ids 5eb02046931446030155d38a 5eb02046931446030155d38b -o C:\Users\Alex\Downloads\annotations -p 5e57b8d248dbe6f07e5f44a0

Or set the following example value in a config file annotations_config_with_slide_ids.yaml and use it as in other calls:

project_id: 5e57b8d248dbe6f07e5f44a0
output_path: C:\Users\Alex\Downloads\annotations
slide_ids:
  - 5eb02046931446030155d38a
  - 5efb4399eaa8022c65f72023

And the command call will look like this:

pyportal.exe da --config annotations_config_with_slide_ids.yaml

As a result of this call, annotations will be placed in the folder 'C:\Users\Alex\Downloads\annotations' in the following structure:

C:\Users\Alex\Downloads\annotations/
├── annotation_set_1
│   └── slide_name_1.geojson
├── annotation_set_2
│   ├── slide_name_1.geojson
│   └── slide_name_2.geojson
├── Validation
│   ├── slide_name_1.geojson
...
└── annotation_set_n
    ├── slide_name_1.geojson
    ├── slide_name_2.geojson

If there are no annotations in the set for one slide, then the file slide_name_xxx.geojson is not created.

Obviously, the 2 above commands can be joined. In that situation only annotations for set name Validation for a slide with the id 5eb02046931446030155d38a will be downloaded:

Example config file annotations_config_with_slide_ids_and_set_name.yaml:

project_id: 5e57b8d248dbe6f07e5f44a0
output_path: C:\Users\Alex\Downloads\annotations
annotation_set: Validation
slide_ids:
    - 5eb02046931446030155d38a

Supplying that config file to PyPortal command, will result in the following structure:

C:\Users\Alex\Downloads\annotations\
└── Validation
    ├── slide_name_1.geojson

Download by class Name or ID

It is possible to download annotations of a specific class name or ID. To filter out region annotations, use the flag --region-class-filter. You can use class ID or class name to filter.

pyportal.exe da -p 5e57b8d248dbe6f07e5f44a0 -o C:\Users\Alex\Downloads\filtered_annotations --region-class-filter "Tumor" "Tumor Area"

Or set the following example value in a config file annotations_config_with_slide_ids.yaml and use it as in other calls:

project_id: 5e57b8d248dbe6f07e5f44a0
output_path: C:\Users\Alex\Downloads\filtered_annotations
region_class_filter:
  - "Tumor"
  - "Tumor Area"

Similarly, cell marker annotations can be downloaded by providing value for the --cma-class-filter option.

pyportal.exe da -p 5e57b8d248dbe6f07e5f44a0 -o C:\Users\Alex\Downloads\filtered_annotations --cma-class-filter "CD68" "Premalignant"

Annotations format

Annotations can be downloaded in 2 formats that are described below. Use a special --format parameter in order to use a specific format. By default, GeoJSON is used.

GeoJSON (Default)

This format exports annotations into GeoJSON file: annotations.geojson

Aperio XML (Deprecated)

This format exports annotations into a XML file, that supports Aperio format: Annotations_Markers.xml

Examples

This example will show how to download annotations using the following example config file.

project_id: 5eeb552c7d677c4d5de9797c
slide_ids:
  - 5eeb5c947d677c227ee97ecb
annotation_set: Intensity Score
output_path: pyportal_intensity_score

A call, together with the output should be similar to the following:

pyportal da --config da_config.yaml

INFO - PyPortal v2.10.0
INFO - Downloading annotations for project 5eeb552c7d677c4d5de9797c...
INFO - Region annotations 1 - 0 downloaded
INFO - CMAs 1 - 6 downloaded
Downloading annotations for 1 slides (current: 5eeb5c947d677c227ee97ecb): 100%|████████████████████████| 1/1 [00:00<00:00,  4.32slide/s]
INFO - 6 annotations downloaded.
INFO - Wrote Annotation GeoJSON file "pyportal_intensity_score/Intensity Score/MDM001_MOS739_RDG_ML1713200_20170928B_FoxP3+PD1+CD8.geojson".

Which will produce a folder structure similar to this:

.
├── da_config.yaml
└── pyportal_intensity_score
    └── Intensity\ Score
        └── MDM001_MOS739_RDG_ML1713200_20170928B_FoxP3+PD1+CD8.geojson

All available options for downloading annotations

Linux use

For advanced Linux use, please refer to this documentation.

Known issues

Download slides

To download slides, use the built-in functionality download-slides or its shortcut version ds.

Download all slides

To download all slides using PyPortal, use the following command:

pyportal.exe ds -p [project_id] -o [PATH_WHERE_SLIDES_SHOULD_BE_DOWNLOADED] --portal-api-key [API_KEY]

All slides from your project will be downloaded by default. If there are slides present from your CP Portal project in the destination folder from a previous download, they will be skipped and not downloaded again. Download will continue from where you left off the previous time.

If a slide has an 'invalid' status (e.g. corrupted file), during the download PyPortal will automatically skip this file and the download will proceed, but with valid files only. Each slide is downloaded into its own, separate folder.

By providing an extra parameter --fast-download you can use new and faster way of downloading slides. Please note however, that this option does not allow to continue where you left off in case of a failure or partially downloaded slide. If this option is used and there is an interruption in the download process, the whole slide will be re-downloaded starting from scratch.

By default, the download option "with continued download" is enabled. Faster download needs to be enabled explicitly by specifying the parameter when issuing the command.

Download a filtered set of slides

1. In the CP Portal filter for the set of slides you want to download

   • Please see articles on a global search and column filtering for more information on how to filter for slides

2. Download the PyPortal config file

   1. Click on the Export button
   2. Click on the PyPortal config option
   
   

   Image: Export drop-down options, with PyPortal config option highlighted

3. Export the config file, by clicking on the Export button in the confirmation dialog

   
   

   Image: Confirmation dialog for exporting a config file when filters are active

4. Download slides using the config file and PyPortal

   pyportal.exe ds --config [PATH_TO_DOWNLAODED_CONFIG] -o [PATH_WHERE_SLIDES_SHOULD_BE_DOWNLOADED]

All available options

Running PyPortal on SCP

To run PyPortal on SCP please refer to this article.

If you are having issues installing PyPortal for Linux, please refer to the PyPortal README documentation.

Download raster annotations

To download raster annotations from a project, use the GxP script's download-raster-annotations command or its shortcut dra.

For information about TIFF format requirements and metadata specifications, see the Raster Annotations Format section.

uv run --locked gxp-scripts/pyportal-cli.py download-raster-annotations --help

Download all raster annotations

To download all raster annotations from a set, provide the project ID, raster annotation set name, and output directory:

uv run --locked gxp-scripts/pyportal-cli.py download-raster-annotations \
    --project-id [PROJECT_ID] \
    --ra-set-name "[ANNOTATION_SET_NAME]" \
    --output-path [OUTPUT_DIRECTORY]

Download from specific slides

To download raster annotations for specific slides only, use the --slide-ids flag:

uv run --locked gxp-scripts/pyportal-cli.py dra \
    --project-id [PROJECT_ID] \
    --ra-set-name "[ANNOTATION_SET_NAME]" \
    --output-path [OUTPUT_DIRECTORY] \
    --slide-ids [SLIDE_ID_1] [SLIDE_ID_2]

uv run --locked gxp-scripts/pyportal-cli.py dra --config [PATH_TO_CONFIG_FILE]

The download process creates:

• ra_metadata.csv - Metadata file containing all annotation information (see metadata requirements)
• TIFF files - Individual raster annotation images as referenced in the metadata (see TIFF layer structure)

All available options

Upload annotations

To upload annotations, use the built-in functionality upload-annotations or its shortcut version ua.

Please note that the annotation upload will only import annotations themselves (polygons, lines, cell markers), but not the associated metadata!

Upload annotations process

To see the instructions in the CLI tool and all available arguments, type:

pyportal.exe ua -h

To upload annotations "as is" into the Portal, two parameters have to be provided

1. Exactly one CP Portal project ID where annotations should be stored to

2. The folder location where annotations are located, or alternatively a path to a specific file containing annotations

   • From the provided folder path only files with format `.xml`, `.annotations`, `.json` or `.geojson` are going to be uploaded.

pyportal.exe ua -p [PORTAL_PROJECT_ID] [FOLDER_PATH_WITH_ANNOTATIONS]

Instead of command line arguments, you can provide a config file instead.

Content of an example config file annotations_config.yaml:

project_id: 5e57b8d248dbe6f07e5f44a0

The example config file, can be used in the following way:

pyportal.exe ua --config annotations_config.yaml [FOLDER_PATH_WITH_ANNOTATIONS]

Mapping of class names

The Portal has a consolidated set of class names for annotations. Furthermore, for cell marker annotations (CMAs), a new concept of labelling cell positions with multiple attributes was introduced (e.g. a cell can be lymphoid, positive for CD8 and negative for CD3 at the same time).

Annotations of type FOV (FOV, HPF, or ROIs) are automatically imported by PyPortal to annotations of class FOV. There is no need to explicitly map them with class mappings such as None: FOV.

All currently existing scripts require cell marker positions in a format where each position can only have one attribute. To support your work with CMAs or legacy algorithms, which still require specific class names for regions, there are different possibilities to map them when downloading annotations.

How to find the annotation class names for mapping?

When using the remapping options, make sure to double-check supported region annotation classes and supported cell marker annotation classes and to use the correct spelling of class names and attributes as used in the CP Portal database. They don't match 1:1 in all cases with the names you can see in the portal GUI. You can also use the Portal GUI to find out the correct class_id for the config file by hovering over the class name:

Image: Class ID of annotation classes presented in the CP Portal GUI

Allowed rating attributes of CMA classes are: po for positive, ne for negative, and un for uncertain.

Allowed intensity score attributes of CMA classes are: 0, 1+, 2+, and 3+.

How to map XML classes to CP Portal classes?

Example 1

In the Portal, a single cell can be annotated with multiple classes and each class can be marked as positive or negative. For this reason, special mapping is possible when uploading annotations.

In that case, the annotations_config.yaml should be extended with the proper mapping separately for CMAs and region annotations.

The example file annotations_config_with_mapping.yaml contains the following data:

 1 project_id: 5e57b8d248dbe6f07e5f44a0
 2
 3 # definitions of CMA mapping
 4 cma_mapping:
 5     Lymphoid Positive (blue):
 6     - class_id: lymphoid
 7         rating: po
 8     - class_id: CD8
 9         rating: po
10     Lymphoid Ambiguous (light blue):
11     - class_id: lymphoid
12         rating: un
13
14 # definitions of region annotations mapping
15 region_mapping:
16 # HALO class: Portal class
17     Other: Excl
18     None: FOV

When using the config file with the above entries the following thing will be done:

• line 1 → annotations will be uploaded into the slides of the project with ID: 5e57b8d248dbe6f07e5f44a0
• lines 4-12 → contains information about CMA mapping

• lines 5-9 → every cell marker with the class Lymphoid Positive (blue) will be mapped to one cell with two attributes lymphoid = positive AND CD8 = positive
• lines 10-12 → every cell marker with the class Lymphoid Ambiguous (light blue) will be mapped to one cell with the attribute lymphoid = uncertain

• lines 14-16 → contains information about region annotations mapping

• line 15 → every region annotation with the class Other will be uplaoded as a class Excluded
• line 16 → every region annotation without a class name (e.g. None) will be uploaded as an annotation with a class name FOV

The PyPortal call should look like this:

pyportal.exe ua --config annotations_config_with_mapping.yaml [FOLDER_PATH_WITH_ANNOTATIONS]

Example 2

The example file annotations_config_with_mapping.yaml contains the following data:

 1 project_id: 5e57b8d248dbe6f07e5f44a0
 2
 3 # definitions of CMA mapping
 4 cma_mapping:
 5     PDL1:
 6     - class_id: CD8
 7         rating: po
 8     FOXP3 positive cells:
 9     - class_id: FOXP3
10         rating: po
11     CD8FOXP3 positive cells:
12     - class_id: CD8
13         rating: po
14     - class_id: FOXP3
15         rating: po

Please note that class names in a mapping must match the text attribute that can be found in the XML file. When mapping values exported from the CP Portal which are being imported also to the CP Portal usually you don't need to use positive cells when the cell is single.

...
    <Region NegativeROA="0" Selected="0" Text="CD8" Type="2">
...

When using the config file with above entries the following thing will be done:

• line 1 → annotations will be uploaded into the slides of project with ID: 5e57b8d248dbe6f07e5f44a0
• lines 5-7 → every cell marker with the class CD8 positive cells will be mapped to one CD8 positive cell
• lines 8-10 → every cell marker with the class FOXP3 positive cells will be mapped to one FOXP3 positive cell
• lines 11-15 → every cell marker with the class CD8FOXP3 positive cells will be mapped to one cell with attributes CD8 positive AND FOXP3 positive

The PyPortal call should look like this:

pyportal.exe ua --config annotations_config_with_mapping.yaml [FOLDER_PATH_WITH_ANNOTATIONS]

Upload to a specific set

To upload only annotations from a specific set only, one can use the flag --annotation-set:

pyportal.exe ua --annotation-set [ANNOTATION_SET_NAME] [FOLDER_PATH_WITH_ANNOTATIONS]

Alternatively, you can provide a config file. Here is an example config file annotations_config_with_set_name.yaml containing the following data:

 1 project_id: 5e57b8d248dbe6f07e5f44a0
 2 annotation_set: Validation
 3
 4 # definitions of CMA mapping
 5 cma_mapping:
 6   CD8 positive cells:
 7    - class_id: CD8
 8      rating: po
 9   FOXP3 positive cells:
10    - class_id: FOXP3
11      rating: po
12   CD8FOXP3 positive cells:
13    - class_id: CD8
14      rating: po
15    - class_id: FOXP3
16      rating: po

You can provide that config file to PyPortal in the following way:

pyportal.exe ua --config annotations_config_with_set_name.yaml [PATH_TO_FOLDER_WITH_ANNOTATIONS]

Rename Cell-Marker Annotation Classes by Down- and Upload

In this example, we show how to change cell-marker annotation (CMA) classes. Sometimes, pathologists create CMAs with wrong classes, and down- and uploading them with a class mapping is a good way to fix this:

• Download cell-markers:
  In the CP Portal, CMAs do not have class names, but a list of class dictionaries with class ID and rating or intensity score. During download to XML and GeoJSON files, PyPortal generates class names based on the class dictionaries.
  Examples: Cell-markers with class CD8 and rating "negative" are downloaded with class name "CD8-ne". Class FOXP3 and intensity score "1+" becomes "FOXP3-1+". "positive" CMAs get no suffix, so a "positive" PD1 CMA becomes just "PD1".
• Upload downloaded CMAs with class mapping:
  During annotation upload, PyPortal allows assigning new Portal class dictionaries based on the class names in the annotation XML or GeoJSON files.
  This example config maps the "positive" CD8 cell-markers ("CD8", no suffix) to "positive" CD68 cell-markers and changes the intensity score of the FOXP3 cell-markers from "1+" ("FOXP3-1+") to "2+":

  project_id: 5eeb552c7d677c4d5de9797c
  annotation_set: Intensity Score
  annotations_path: pyportal_intensity_score_mapping
  
  cma_mapping:
    CD8:
      - class_id: CD68
        rating: po
    FOXP3-1+:
      - class_id: FOXP3
        intensity_score: 2+

All available options

Linux use

For an advanced usage in Linux please refer to this article.

Upload landmarks

Obtain landmarks

The co-registration tool Slidematch is used for generating landmark files in a JSON format.

Please reach out to Slidematch team if you are not sure about the usage of that program.

We propose to use the Portal-generated IDO file with the slides that you want to generate landmarks for. This way, all information about slide file name, stains (assay info) and location in the file share will be propagated to the landmarks JSON file and can be automatically linked to the slides in Portal.

Upload landmarks to a project

To upload landmarks, use the built-in functionality upload-landmarks or its shortcut version ul.

To see the instructions in the CLI tool and all available arguments, type:

pyportal.exe ul -h

Landmarks can be uploaded by providing the Project ID to which landmarks should be uploaded, and the path to a folder which contains landmark files.

pyportal.exe ul -p [PROJECT_ID] --landmarks-path [PATH_TO_FOLDER_WITH_LANDMARKS]

All available options

Extend landmarks

Extend existing landmarks

Landmarks are extended by adding new stains present in given JSON files and not present yet in the Portal. Landmark positions on newly added stains are calculated by transfering positions on common stains, so it is required that at least one stain in given JSON files is already present in the Portal. The existing stains and landmark positions in the Portal are left untouched.

Be aware, that landmarks created by extending existing landmarks with new stains are usually a bit different than landmarks created by registering all stains at once in Slidematch and they may cause slightly different alignment of the slides.

To extend already existing landmarks, use the built-in functionality extend-landmarks or its shortcut version el.

To see the instructions in the CLI tool and all available arguments, type:

pyportal.exe el -h

Landmarks can be extended by providing the Project ID to which landmarks should be uploaded, and the path to a folder which contains landmark files.

pyportal.exe el -p [PROJECT_ID] --landmarks-path [PATH_TO_FOLDER_WITH_LANDMARKS]

All available options

Upload slides

Upload slides from a local drive

Slides can be uploaded to the CP Portal using PyPortal's Upload Slides functionality. Typically, this method results in faster upload speeds and less time overall, compared to uploading slides via the browser.

To upload slides, use the built-in functionality upload-slides or its shortcut version us.

To see the instructions in the CLI tool and all available arguments, type:

pyportal.exe us -h

Slides can be uploaded by providing the Project ID to which slides should be uploaded, and the path to a folder which contains digital pathology slide files.

pyportal.exe us -p [PROJECT_ID] -s [PATH_WHERE_SLIDES_ARE_LOCATED]

Here is an example run of uploading slides to the CP Portal using the PyPortal's uplad-slides functionality.

pyportal us -p 62625f998d52266f327dba7c -s ./input
INFO - PyPortal v2.23.0
INFO - Collected slides: 2
INFO - Slide 64e5a269f8f12a1f6b387208 creation was initiated.
INFO - Slide 64e5a269f8f12a1f6b387208 was added to the project: 62625f998d52266f327dba7c
INFO - Slide 64e5a269f8f12a1f6b387208 is already available, skipping.
INFO - Slide 64e5a269f8f12a1f6b387209 creation was initiated.
INFO - Slide 64e5a269f8f12a1f6b387209 has status: new.
input/pyportal_upload_test_22.tif: 100%|█████████████████████████████████████████████████| 16.3M/16.3M [04:03<00:00, 70.3kB/s]
INFO - Collected 2 slides, uploaded: 1, failed: 0, skipped: 1.
INFO - 16.3 MB uploaded in 0:04:05.414386 seconds. Speed: 0.07 MB/s.

All available options

Upload raster annotations

Raster annotations are image-based annotations that can be uploaded using PyPortal's GxP-compliant CLI script. This functionality allows for bulk operations with raster annotation data, including version tracking and provenance management.

The GxP script is located in the PyPortal repository at gxp-scripts/pyportal-cli.py and uses the uv package manager.

For TIFF format requirements and metadata specifications, see the Raster Annotations Format section.

uv run --locked gxp-scripts/pyportal-cli.py upload-raster-annotations --help

Upload process

To upload raster annotations to a project, the source directory used for the upload must contain:

• ra_metadata.csv file with metadata for each annotation (see metadata requirements)
• TIFF files with the raster annotation data (see TIFF layer structure)

Basic upload command:

uv run --locked gxp-scripts/pyportal-cli.py upload-raster-annotations \
    --project-id [PROJECT_ID] \
    --ra-set-name "[ANNOTATION_SET_NAME]" \
    --source-directory [SOURCE_DIRECTORY]

All available options

Raster Annotations Format

This section describes the technical format requirements for raster annotation TIFF files and metadata.

TIFF layer structure

Raster annotation TIFF files must contain exactly 6 channels in the following order. All files must use uint32 data type with background value 0.

Metadata requirements

The ra_metadata.csv file must contain the following columns. The source directory should also contain the TIFF files referenced by raster_annotation_filename.

Required columns (cannot be empty):

Conditionally required columns:

Optional columns:

• review_status - Workflow status: "toAnnotate", "toReview", "accepted", etc.
• raster_annotation_set_description - Human-readable description of the set
• raster_annotation_set_experiment_description - Description of the experiment

Example image_metadata value:

{'Regions': {'class_map': {'1': {'name': 'Tumor', 'color': '0xa31ae5'}}}}