Introduction to workflows
Workflows are meant to guide a team of users through a certain repetitive processes consisting of certain steps. A task management system is provided as the single entry point to perform actions in the CP Portal. To avoid accidental changes the workflows automize repeating steps and users only get access to the project’s data when they need to make changes. Additionally users are guided through tailored UI views to simplify their tasks.
Workflow user role
Any user that should work within a workflow should be assigned the workflow user role. Users with this role have a limited view on the Portal’s UI. Their only entry point to change data in a project is through tasks that are assigned to them through a workflow.
Important
- Technically workflow users have write access to a project. While the UI pevents them from executing these write permissions to objects and data outside their tasks, they have full write access to a project through the APIs
Task list
The task list shows all pending tasks a user has in a
project. Also using the link provided with the task allows
the user to jump to the view needed to carry out the
actions to complete the task. Some tasks can be closed
through the task list, others can only be completed from
within the task specific view after using the provided
link.
The search field (1) lets you filter the task list. When
you enter a search term, only tasks with slides whose
metadata fields contain that term will be displayed.
Image: Workflow task list with pending tasks
Workflow types
Test Set Annotation Workflow
The Test Set Annotation workflow serves to create test set annotations. This requires a lead pathologist to place FOVs on a set of slides to cover all required combinations of morphological features. Additionally he places ROIs in the FOVs and CMAs. This is referred to as the template. The FOVs are then going into three parallel streams where several groups of pathologists provide raster annotations for ART. The groups all receive the same FOVs, in order to even out slight differences between the annotators. Once an ART annotation task is completed a SSTC task follows where the ROI and the CMAs of the lead pathologist are provided. At the end of the workflow each group has provided ART and SSTC annotations for all templates
Train Set Annotation Workflow
The Train Set Annotation workflow serves to create train set annotations. This is done model performance based, meaning a model, that is supposed to be improved, needs to be run as part of the workflow. A lead pathologist then places FOVs on a set of slides to cover areas that the model did not perform well on. The FOVs are then going to be equally split among the assigned pathologists who provide raster annotations for ART or SSTC depending on the configuration of the workflow.
Validation Set Annotation (Morphology based) Workflow
The Validation Set Annotation (Morphology based) workflow serves to create validation annotations. This requires a lead pathologist to place FOVs on a set of slides to cover all required combinations of morphological features.. The FOVs are then going to be equally split among the assigned pathologists who provide raster annotations for ART or SSTC depending on the configuration of the workflow.
Validation Set Annotation (Model-performance based) Workflow
The Validation Set Annotation (Model-performance based) workflow serves to create validation annotations. This is done model performance based, meaning a model, needs to be run as part of the workflow. A lead pathologist then places FOVs on a set of slides depending on the performance of the model. The FOVs are then going to be equally split among the assigned pathologists who provide raster annotations for ART or SSTC depending on the configuration of the workflow.
Whole Slide Image (WSI) Screening Workflow
The WSI Screening workflow is designed for morphological assessment of slides without requiring annotation. This workflow enables systematic review and screening of whole slide images for quality control or initial evaluation purposes.
Review Annotation Workflow
The Review Annotation Workflow is meant for qualifying existing raster annotation sets. For this it allows existing raster annotations to be reviewed, if necessary corrected and finally signed off.
FOV Placement Workflow
The FOV Placement Workflow is designed to guide users through the selection and placement of fields of view (FOVs) on project slides. Depending on the workflow configuration, users can create Unified FOV Annotations, ART Annotations, or SSTC Annotations. The workflow can optionally include a solution run before FOV placement, create annotation sets automatically, assign FOV placement tasks to one or more users, and support structured FOV tracking based on configured metadata targets.
Raster Annotation Workflow
The Raster Annotation Workflow is designed to create raster annotations for an existing set of fields of view (FOVs). It takes the FOVs from a selected annotation set and turns them into annotation tasks so that pathologists can provide ART, SSTC, or Unified annotations. Depending on the configuration, the workflow can use multi-pathologist or single-pathologist annotation modes and optionally provide an existing or newly created solution result (DRC) as a starting point for the annotations.
Note: ResultQC workflow was removed
- ResultQC workflow was removed with version 4.3.0 of CP Portal and is no longer available. It is planned to replace it with a similar workflow in the future.
General Workflow Configuration
By opening the workflow configuration (1) a link to a work instruction document can be entered (2). This link will then be available throughout all tasks that are generated as part of the configured workflow
Image: Workflow instructions configuration
Clone workflow configuration
Once a workflow has been started, its configuration becomes fixed and cannot be edited to ensure consistency and traceability. If you need to modify a workflow that has already been started, you must clone it and save it under a new name. This ensures that the original workflow configuration remains intact for reference while allowing you to create and customize a new version.
Image: Clone button in Workflow Manager
Configure Test Set Annotation Workflow
Setup Test Set Annotation Workflow configuration
To set up a new Test Set Annotation Workflow, drag and drop the Test Set Annotation Workflow template into the workflow manager. The workflow manager is automatically populated with a chain of the following steps:
- Slide selection step
- Create Annotation Set step
- Create Test Set Template step
- Configure Annotation step
- Annotate ART and SSTC step
Image: Workflow manager with Test Set Annotation Workflow
Slide selection configuration
As part of this step the slides that will be fed in the workflow can be defined. By default inclusion of all slides present in the project at the start of the workflow is selected. As an alternative, the box can be unchecked and the step can be configured to read a filter link, so all slides that fullfill the conditions of the filter link at the start of the workflow will be fed in the workflow.
Image: Slide selection configuration window
Note: Task creation in Test set Annotation Workflow is based on slides
- In the Test Set Annotation Workflow a task is created for every slide that is provided. Supporting H&E slides of the same block are automatically shown as part of the workflow and should not need be explicitly included in the filter link.
Create Annotation Sets Configuration
This step automatically creates annotation sets for all slides in the workflow. The name for the annotation set that should be generated has to be specified in the workflow step’S configuration.
Image: Run solution configuration window
Create Test Set Template Configuration
First a lead pathologist for creating the template (placement of FOVs, ROIs and CMAs) can be selected. Secondly the FOV placement tracker can be configured. It allows to set up a combination of metadata fields and their CV entries as well as a target of how many FOVs with the respective combination of characteristics should be part of the final set of FOVs.
Image: Create Test Set Template configuration window
Note: Counting of FOV placement tracker
- The system will count an FOV that is placed on a slide with the given slide metadata CV entry or in case of annotation metadata, if the FOV metadata shows a configured annotation metadata entry. When several conditions are met, the FOV counts once for each codition.
Configure Annotation Configuration
In this step the settings for creation of the raster annotations are made. For each group of annotators, users can be selected. Each group needs at least one annotator assigned to it and no annotator can be part of two different groups. Apart from that the annotation classes that shall be available for the ART annotation step can be selected. Lastly, the annotation set base names for the annotation sets for ART and SSTC can be selected.
Image: Configure Annotation window
Note: Annotation Set Group number
- During the workflow , each annotation set name is appended with the group numbers 1 to 3 automatically to allow distinguishing them.
Annotate ART and SSTC configuration
This step does not require any configuration.
Configure Train Set Annotation Workflow
Setup Train Set Annotation Workflow configuration
To set up a new Train Set Annotation Workflow, drag and drop the Train Set Annotation Workflow template into the workflow manager. The workflow manager is automatically populated with a chain of the following steps:
- Slide selection step
- Run solution step
- Create Annotation Set step
- Place FOVs step
- Configure Annotation step
- Annotate ART or SSTC step
Image: Workflow manager with Train Set Annotation Workflow
Slide selection configuration
As part of this step the slides that will be fed in the workflow can be defined. By default inclusion of all slides present in the project at the start of the workflow is selected. As an alternative, the box can be unchecked and the step can be configured to read a filter link, so all slides that fulfil the conditions of the filter link at the start of the workflow will be fed in the workflow.
Image: Slide selection configuration window
Note: Task creation in Train Set Annotation Workflow is based on slides
- In the Train Set Annotation Workflow a task is created for every slide that is provided. Supporting H&E slides of the same block are automatically shown as part of the workflow and do not need to be explicitly added in the filter link.
Run Solution configuration
This step is meant to automatically execute a Predictino pipeline with a configured model on all slides in the workflow. The delivery created by the pipeline will be automatically available in the view for placing FOVs when opening a task to “Place FOVs” from the task list. The configuration steps depend on the selected model and are the same as when running Predictino from the UI. The only difference is that the configuration is saved and the job is only started once the workflow reaches the “Run Solution” step.
Important
- Some models generate several deliveries. The system selects the delivery that matches the experiment name configured as part of the Run Solution configuration available in the view for Place FOVs.
Image: Run solution configuration window
Create Annotation Sets Configuration
This step automatically creates annotation sets for all slides in the workflow. The name for the annotation set that should be generated has to be specified in the workflow step’S configuration.
Image: Create Annotation Sets configuration window
Place FOVs Configuration
In this configuration the lead pathologist for placing the FOVs can be selected.
Image: Place FOVs configuration window
Configure Annotation Configuration
In this step the settings for creation of the raster annotations are made. The annotators among which the system equally shares the annotation tasks can be selected as can be the pool of reviewers for the review of the raster annotations. Additionally, it can be configured if the annotators are supposed to create ART or SSTC annotations and a name for the raster annotation set that will be created to hold all raster annotations created as part of the workflow. In case ART annotations are configured, the available ART annotation classes can be selected as well.
Image: Configure Annotation window
Annotate ART and SSTC configuration
This step does not require any configuration.
Configure Validation Set Annotation (Morphology based) Workflow
Setup Validation Set Annotation (Morphology based) Workflow configuration
To set up a new morphology based Validation Set Annotation Workflow, drag and drop the Validation Set Annotation Workflow template into the workflow manager. The workflow manager is automatically populated with a chain of the following steps:
- Slide selection step
- Create Annotation Set step
- Place FOVs step
- Run Solution step
- Configure Annotation step
- Annotate ART or SSTC step
Image: Workflow manager with morphology based Validation Annotation Workflow
Slide selection configuration
As part of this step the slides that will be fed in the workflow can be defined. By default inclusion of all slides present in the project at the start of the workflow is selected. As an alternative, the box can be unchecked and the step can be configured to read a filter link, so all slides that fullfill the conditions of the filter link at the start of the workflow will be fed in the workflow.
Image: Slide selection configuration window
Note: Task creation in Validation Set Annotation Workflows is based on slides
- In the Validation Set Annotation Workflows a task is created for every slide that is provided. Supporting H&E slides of the same block are automatically shown as part of the workflow and should not need be explicitly included in the filter link.
Create Annotation Sets Configuration
This step automatically creates annotation sets for all slides in the workflow. The name for the annotation set that should be generated has to be specified in the workflow step’S configuration.
Image: Run solution configuration window
Place FOVs Configuration
First a lead pathologist for creating the template (placement of FOVs, ROIs and CMAs) can be selected. Secondly the FOV placement tracker can be configured. It allows to set up a combination of metadata fields and their CV entries as well as a target of how many FOVs with the respective combination of characteristics should be part of the final set of FOVs.
Image: Place FOVs configuration window
Note: Counting of FOV placement tracker
- The system will count an FOV that is placed on a slide with the given slide metadata CV entry or in case of annotation metadata, if the FOV metadata shows a configured annotation metadata entry. When several conditions are met, the FOV counts once for each codition.
Run Solution configuration
This step is meant to automatically execute a Predictino pipeline with a configured model on all slides in the workflow. The delivery created by the pipeline will be automatically available in the view for placing FOVs when opening a task to “Place FOVs” from the task list. The configuration steps depend on the selected model and are the same as when running Predictino from the UI. The only difference is that the configuration is saved and the job is only started once the workflow reaches the “Run Solution” step.
Important
- Some models generate several deliveries. The system selects the delivery that matches the experiment name configured as part of the Run Solution configuration available in the view for Place FOVs.
Image: Run solution configuration window
Configure Annotation Configuration
In this step the settings for creation of the raster annotations are made. The annotators among which the system equally shares the annotation tasks can be selected as can be the pool of reviewers for the review of the raster annotations. Additionally, it can be configured if the annotators are supposed to create ART or SSTC annotations and a name for the raster annotation set that will be created to hold all raster annotations created as part of the workflow. In case ART annotations are configured, the available ART annotation classes can be selected as well.
Image: Configure Annotation window
Annotate ART or SSTC configuration
This step does not require any configuration.
Configure Validation Set Annotation (Model-performance based) Workflow
Setup Validation Set Annotation (Model-performance based) Workflow configuration
To set up a new Validation Set Annotation (Model-performance based) Workflow, drag and drop the corresponding template into the workflow manager. The workflow manager is automatically populated with a chain of the following steps:
- Slide selection step
- Run solution step
- Create Annotation Set step
- Place FOVs step
- Configure Annotation step
- Annotate ART or SSTC step
Image: Workflow manager with Validation Set Annotation (Model-performance based) Workflow
Slide selection configuration
As part of this step the slides that will be fed in the workflow can be defined. By default inclusion of all slides present in the project at the start of the workflow is selected. As an alternative, the box can be unchecked and the step can be configured to read a filter link, so all slides that fulfil the conditions of the filter link at the start of the workflow will be fed in the workflow.
Image: Slide selection configuration window
Note: Task creation in Validation Set Annotation Workflows is based on slides
- In the Validation Set Annotation Workflows a task is created for every slide that is provided. Supporting H&E slides of the same block are automatically shown as part of the workflow and do not need to be explicitly added in the filter link.
Run Solution configuration
This step is meant to automatically execute a Predictino pipeline with a configured model on all slides in the workflow. The delivery created by the pipeline will be automatically available in the view for placing FOVs when opening a task to “Place FOVs” from the task list. The configuration steps depend on the selected model and are the same as when running Predictino from the UI. The only difference is that the configuration is saved and the job is only started once the workflow reaches the “Run Solution” step.
Important
- Some models generate several deliveries. The system selects the delivery that matches the experiment name configured as part of the Run Solution configuration available in the view for Place FOVs.
Image: Run solution configuration window
Create Annotation Sets Configuration
This step automatically creates annotation sets for all slides in the workflow. The name for the annotation set that should be generated has to be specified in the workflow step’S configuration.
Image: Create Annotation Sets configuration window
Place FOVs Configuration
In this configuration the lead pathologist for placing the FOVs can be selected.
Image: Place FOVs configuration window
Configure Annotation Configuration
In this step the settings for creation of the raster annotations are made. The annotators among which the system equally shares the annotation tasks can be selected as can be the pool of reviewers for the review of the raster annotations. Additionally, it can be configured if the annotators are supposed to create ART or SSTC annotations and a name for the raster annotation set that will be created to hold all raster annotations created as part of the workflow. In case ART annotations are configured, the available ART annotation classes can be selected as well.
Image: Configure Annotation window
Annotate ART and SSTC configuration
This step does not require any configuration.
Configure WSI Screening Workflow
Setup WSI Screening Workflow configuration
To set up a new WSI Screening workflow, drag and drop the WSI Screening workflow template into the workflow manager. The workflow manager is automatically populated with a chain of the following steps:
- Slide selection step
- Create Annotation Set step
- Perform WSI Screening step
Image: Workflow manager with WSI Screening Workflow
Slide selection configuration
As part of this step the slides that will be fed in the workflow can be defined. By default inclusion of all slides present in the project at the start of the workflow is selected. As an alternative, the box can be unchecked and the step can be configured to read a filter link, so all slides that fullfill the conditions of the filter link at the start of the workflow will be fed in the workflow.
Image: Slide selection configuration window
Note: Task creation in WSI Screening workflows is based on Block IDs
- In the WSI Screening Workflows tasks are on the level of the Block IDs that are found among the provided slides. At a minimum a Block ID must have one H&E image for a task to be created. If additionally at least one IHC image from the Block ID is available and the IHC evluation has been enbaled, a task for IHC assessment can be created as well.
Create Annotation Sets Configuration
This step automatically creates annotation sets for all slides in the workflow. The name for the annotation set that should be generated has to be specified in the workflow step’S configuration.
Image: Run solution configuration window
Perform WSI Screening
The task mode can be selected to decide whether task should be created for HE assessment or IHC assessment or both. Additionally one or several users can be configured to carry out the assessment. If more than one user is selected, the tasks get equally distributed among the configured users.
Image: Perform WSI Screening configuration window
Configure Review Annotation Workflow
Setup Review Annotation Workflow configuration
To set up a new Review Annotation workflow, drag and drop the Review Annotationworkflow template into the workflow manager. The workflow manager is automatically populated with a chain of the following steps:
- Slide selection step
- Select Annotation Set step
- Configure Annotation Review step
- Review ART or SSTC
- Correct ART or SSTC
Image: Workflow manager with Review Annotation Workflow
Slide selection configuration
As part of this step the slides that will be fed in the workflow can be defined. By default inclusion of all slides present in the project at the start of the workflow is selected. As an alternative, the box can be unchecked and the step can be configured to read a filter link, so all slides that fullfill the conditions of the filter link at the start of the workflow will be fed in the workflow.
Image: Slide selection configuration window
Note: Task creation in Review Annotation workflows is based on raster annotation sets.
- Task creation in Review Annotation workflows is based on raster annotation sets. For all raster annotation sets with annotations of the configured mode (ART or SSTC) that are included in the FOVs in the defined vector annotation set on the provided slides, a task is created.
Select Annotation Sets Configuration
This step automatically creates annotation sets for all slides in the workflow. The name for the annotation set that should be generated has to be specified in the workflow step’S configuration.
Image: Select Annotation Set configuration window
Configure Annotation Review
The steps allows for the configuration of reviewers for the annotations. If several reviewers are selected, the tasks get equally distributed among all reviewers. Additionally the desired model type (ART or SSTC) needs to be specified. The FOV random sample size is an optional configuration. It allows to have the system randomly select a specified number of FOVs from the configured annotation set on the provided slides and only create tasks for the raster annotations included in these randomly selected FOVs. Lastly the configuration of annotators allows to restrict the ability of reviewers to request annotation corrections only from the specified users instead of from all eligible users in the project.
Image: Configure Annotation Review configuration window
Configure Review ART or SSTC
This step does not require any configuration.
Correct ART or SSTC
This step does not require any configuration.
Configure FOV Placement Workflow
Setup FOV Placement Workflow configuration
To set up a new FOV Placement workflow, drag and drop the FOV Placement workflow template into the workflow manager. The workflow manager is automatically populated with a chain of the following steps:
- Slide selection step
- Run solution step
- Create Annotation Set step
- Place FOVs step
Image: Workflow manager with FOV Placement Workflow
Slide selection configuration
As part of this step, the slides that will be included in the workflow can be defined. By default, all slides present in the project at the start of the workflow are included. As an alternative, this option can be unchecked and the step can be configured to use a filter link, so that all slides matching the conditions of the filter link at the start of the workflow are included.
Image: Slide selection configuration window
Run solution configuration
This step can be used to automatically execute a configured solution on all slides in the workflow before FOV placement begins. If a delivery is created by the solution, it can be made available during the Place FOVs step and will also be written to annotation metadata as the FOV/ROI placement delivery. If no delivery is needed for the FOV palcement, this step can be configurred to be skipped.
Image: Run solution configuration window
Image: Solution configuration window
Create Annotation Set configuration
This step automatically creates annotation sets for all slides in the workflow. The name for the annotation set to be generated must be specified in the workflow step configuration.
Image: Create Annotation Set configuration window
Place FOVs configuration
In this step, the main settings for FOV placement are defined. One or more users can be assigned to place FOVs or create the template content, depending on the configured annotation workflow type. If more than one user is selected, the system distributes tasks evenly across all configured users.
Users can configure the annotation workflow type as Unified FOV Annotations, ART Annotations, or SSTC Annotations, depending on whether the Run solution step is skipped. In addition, an FOV placement tracker can be configured. The FOV placement tracker allows to set up a combination of metadata fields and their CV entries as well as a target of how many FOVs with the respective combination of characteristics should be part of the final set of FOVs.
If a DRC is configured in the Run solution step and the selected annotation workflow type is ART Annotations or SSTC Annotations, the configuration can define whether DRC is available during FOV placement.
Image: Place FOVs configuration window
Note: Counting of FOV placement tracker
- The system will count an FOV that is placed on a slide with the given slide metadata CV entry or in case of annotation metadata, if the FOV metadata shows a configured annotation metadata entry. When several conditions are met, the FOV counts once for each codition.
Place FOVs tasks
The Place FOVs task opens a workflow-specific viewer for placing and reviewing annotations on slides.
During FOV placement, users can request a secondary review. In this case, the system prompts the user to select another project member. A secondary review task is then created for that user. The original FOV placement task cannot be finalized until the secondary review task has been closed or cancelled.
Users can also reject a slide for quality reasons. In this case, at least one reason must be provided. In addition, the user must enter their e-signature PIN. After that, the Place FOV task is closed.
Annotation metadata during FOV placement
When a user places an FOV, the system automatically populates annotation metadata based on slide-level metadata and workflow configuration where possible.
If editable annotation metadata fields do not have values that can be imported from the slide, they are shown as a greyed-out not provided. This also applies to FOVs that already existed in the annotation set before the workflow task was opened. When the user finalizes the task, any fields still showing the greyed-out not provided are saved as not provided.
Configure Raster Annotation Workflow
Setup Raster Annotation Workflow configuration
The Raster Annotation Workflow is used to create raster annotations for an existing set of Fields of View (FOVs). It takes the FOVs that are present in a selected annotation set and turns them into annotation tasks so that pathologists can provide ART and/or SSTC raster annotations. Depending on the configuration, an existing or newly created delivery (DRC) can be provided as a starting point for the annotations.
To set up a new Raster Annotation workflow, drag and drop the Raster Annotation workflow template into the workflow manager. The workflow manager is automatically populated with a chain of the following steps:
- Slide selection step
- Select Annotation Set step
- Run Result step
- Annotation configuration step
- Annotate ART and/or Annotate SSTC step
Image: Workflow manager with Raster Annotation Workflow
Slide selection configuration
As part of this step, the slides that will be included in the workflow can be defined. By default, all slides present in the project at the start of the workflow are included. As an alternative, this option can be unchecked and the step can be configured to use a filter link, so that all slides matching the conditions of the filter link at the start of the workflow are included.
Select Annotation Set configuration
In this step the annotation set that is fed into the workflow is selected from all annotation set names that are available in the project. The FOVs that are present in the selected annotation set on the slides selected in the Slide selection step are used to create the annotation tasks.
Note: Task creation in Raster Annotation workflows is based on FOVs.
- Task creation in Raster Annotation workflows is based on the FOVs on the configures slides in the selected annotation set. In case of unified annotations, the system creates a task for each FOV and checks at the end of the ART task if there is a valid ROI for SSTC annotations that can be used to create a consequent SSTC annotation task.
When an annotation set is selected, the system checks whether the annotation set exists on all slides. If the annotation set is missing on some slides, the system lists the first up to five slide file names where the annotation set does not exist and shows the total number of additional slides where it is missing.
Run solution configuration
The Run Solution step determines whether a delivery is used as a starting point for the raster annotations.
It is possible to configure that running a solution is skipped. If running a solution is skipped, an existing Delivery (DRC) from the project can be selected. When a delivery is selected, the system checks whether the delivery is available on all slides that are configured to be in the workflow. If the delivery is missing on some slides, the system lists the first up to five slide names where the DRC is missing and shows the total number of additional slides where it is missing. Selecting a delivery is optional; the step can also be saved without selecting one.
If running a solution is not skipped, a solution can be configured to run during the workflow. The result created by that solution is then used in the following ART and SSTC annotation steps.
Image: Run solution configuration window
Image: Solution configuration window
Image: Option to select existing delivery when run solution is skipped
Annotation configuration
In this step the settings for the raster annotations are made. The following options can be configured:
- Annotation workflow type – Test Set, Training Set or Tuning Set. This selection is used for validating the correct configuration of the workflow.
- Annotation mode – multi-pathologist annotations or single-pathologist annotations. This option determines whether each FOV is annotated several times independently by several pathologists without a mandatory review or once with a mandatory review.
- Annotation usage – ART, SSTC, or Unified annotations. This option is to configure if annotations are created in RACT ART, RACT SSTC or both sequentially
- Whether SAM (Segment Anything for proposing annotations) is available in the workflow.
For multi-pathologist annotations, two, three or four annotator groups can be configured. Each user can be part of one group only, and at least two properly configured groups are required. Tasks are split equally among all users in a group.
For single-pathologist annotations, one or several annotators and one or several reviewers can be selected. The configuration must allow every annotator to request the review from a user different from themselves.
When the annotation usage is set to ART, the ART annotation classes that shall be made available during the Annotate ART step can be selected. At least one ART annotation class must be selected.
The raster annotation set names are pre-filled based on the selected vector annotation set name, but can be adapted by the user.
Image: Configure annotation step of raster annotation workflow
Allowed configuration combinations
The available annotation modes and the availability of a delivery (DRC) depend on the combination of annotation workflow type and annotation usage. The following table lists the allowed combinations. Each cell shows the annotation modes that can be selected and whether a delivery may be configured for that combination.
| Annotation usage \ Workflow type | Test Set | Training Set | Tuning Set |
|---|---|---|---|
| ART |
Multi-pathologist or single-pathologist DRC: not available |
Multi-pathologist or single-pathologist DRC: available (single-pathologist only) |
Multi-pathologist or single-pathologist DRC: available (single-pathologist only) |
| SSTC |
Multi-pathologist or single-pathologist DRC: not available |
Multi-pathologist or single-pathologist DRC: available (single-pathologist only) |
Multi-pathologist or single-pathologist DRC: available (single-pathologist only) |
| Unified annotations |
Multi-pathologist or single-pathologist DRC: not available |
Multi-pathologist or single-pathologist DRC: available (single-pathologist only) |
Multi-pathologist or single-pathologist DRC: available (single-pathologist only) |
Annotate ART and/or Annotate SSTC configuration
This step does not require any configuration.
Save and start a workflow
You can save your workflow configuration by using the “save” button and entering a name for the workflow. After that you can retrieve the workflow from the drop-down above the templates.
To start a workflow all steps need to be configured. Also no other workflow is allowed to be running in the same project.
To start the workflow click the “Start” button.