Command Reference

copy

Example

    corgie copy \
--src_layer_spec '{
   "name": "unaligned",
   "path": "gs://corgie/demo/example_stack/img/unaligned"
   }' \
--src_layer_spec '{
   "name": "fold_mask",
   "type": "mask",
   "path": "gs://corgie/demo/example_stack/mask/fold_mask"
   }' \
--dst_folder $CORGIE_WALKTHROUGH_PATH \
--mip 6 \
--start_coord "150000, 150000, 17000" \
--end_coord "200000, 200000, 17010" \
--chunk_xy 1024 --chunk_z 1

Documentation

corgie copy

Copy a bounding box including masks.

corgie copy [OPTIONS]

Options

-s, --src_layer_spec <src_layer_spec>

Required Source layer spec. Use multiple times to include all masks, fields, images. [required]

Format: JSON string.

Optional keys:

“type”: str from [‘img’, ‘segmentation’, ‘field’, ‘mask’, ‘section_value’, ‘fixed_field’, ‘float_tensor’] ;

“name”: str, DEFAULT – same as layer type. Required if given more than 1 layer of the same type ;

“data_backend”: str from [‘cv’, ‘json’] ;

“args”: a dictionary with additional layer parameters,

eg binarization scheme for masks data_mip_ranges, etc;

Required keys: “path”.

--dst_folder <dst_folder>: Destination folder for the copied stack

-c, --chunk_xy <chunk_xy>

Default: 1024

--chunk_z <chunk_z>

Default: 1

--mip <mip>: Required

--blackout_masks, --no_blackout_masks

Default: False

--copy_masks, --no_copy_masks

Default: True

--force_chunk_xy <force_chunk_xy>: Will force the chunking of the underlying cloudvolume

--force_chunk_z <force_chunk_z>: Will force the chunking of the underlying cloudvolume

--start_coord <start_coord>: Required

--end_coord <end_coord>: Required

--coord_mip <coord_mip>

Default: 0

--suffix <suffix>

-q, --queue_name <queue_name>: Name of the queue where the tasks will be pushed to. For file queue, use ‘fq://{path_to_shared_storage}’ format.If no ‘fq://’ prefix is given, the queue is assumed to be SQS. If not specified, tasks will be executed locally.

--completion_queue_name <completion_queue_name>: Name of AWS SQS queue where completion of tasks will be reported. Must be distinct from the ‘queue_name’. Providing a completion queue will speed up execution when running workers on unreliable machines (preemptible, spot).

--sqs_queue_region <sqs_queue_region>: AWS region of SQS queues. Task queue and completion queue must share the same region.

--restart_from_checkpoint_file <restart_from_checkpoint_file>: Option to be used in case of a job interruption. Specify path to filename where job progress is logged so that execution can start from where it left off before the interruption.

-d, --device <device>

Pytorch device specification. Eg: ‘cpu’, ‘cuda’, ‘cuda:0’

Default: cpu

-v, --verbose: Turn on debug logging

downsample

Example

corgie downsample \
--src_layer_spec '{
   "path": "'${CORGIE_WALKTHROUGH_PATH}'/img/unaligned"
   }' \
--mip_start 6 --mip_end 8 \
--start_coord "150000, 150000, 17000" \
--end_coord "200000, 200000, 17010" \
--chunk_xy 1024

Documentation

corgie downsample

Create additional image pyramid layers (mips).

Unlike with copy command, the destination path parameter is not mandatory and will default to the source layer path.

Downsampling for the mask and the image layer have to be done separately, because images are downsampled with average pooling strategy while masks are downsampled with max pooling strategy.

corgie downsample [OPTIONS]

Options

-s, --src_layer_spec <src_layer_spec>

Required Specification for the source layer. [required]

Format: JSON string.

Optional keys:

“type”: str from [‘img’, ‘segmentation’, ‘field’, ‘mask’, ‘section_value’, ‘fixed_field’, ‘float_tensor’] ;

“name”: str, DEFAULT – same as layer type. Required if given more than 1 layer of the same type ;

“data_backend”: str from [‘cv’, ‘json’] ;

“args”: a dictionary with additional layer parameters,

eg binarization scheme for masks data_mip_ranges, etc;

Required keys: “path”.

-s, --dst_layer_spec <dst_layer_spec>: Specification for the destination layer. Refer to ‘src_layer_spec’ for parameter format. DEFAULT: Same as src_layer_spec

-m, --mip_start <mip_start>: Required

-e, --mip_end <mip_end>: Required

-c, --chunk_xy <chunk_xy>

Default: 2048

--chunk_z <chunk_z>

Default: 1

--mips_per_task <mips_per_task>

Default: 3

--preserve_zeros, --no_preserve_zeros

Default: False

--start_coord <start_coord>: Required

--end_coord <end_coord>: Required

--coord_mip <coord_mip>

Default: 0

-q, --queue_name <queue_name>: Name of the queue where the tasks will be pushed to. For file queue, use ‘fq://{path_to_shared_storage}’ format.If no ‘fq://’ prefix is given, the queue is assumed to be SQS. If not specified, tasks will be executed locally.

--completion_queue_name <completion_queue_name>: Name of AWS SQS queue where completion of tasks will be reported. Must be distinct from the ‘queue_name’. Providing a completion queue will speed up execution when running workers on unreliable machines (preemptible, spot).

--sqs_queue_region <sqs_queue_region>: AWS region of SQS queues. Task queue and completion queue must share the same region.

--restart_from_checkpoint_file <restart_from_checkpoint_file>: Option to be used in case of a job interruption. Specify path to filename where job progress is logged so that execution can start from where it left off before the interruption.

-d, --device <device>

Pytorch device specification. Eg: ‘cpu’, ‘cuda’, ‘cuda:0’

Default: cpu

-v, --verbose: Turn on debug logging

normalize

Example

corgie normalize \
--src_layer_spec '{
   "path": "'${CORGIE_WALKTHROUGH_PATH}'/img/unaligned",
      "name": "img"
      }' \
--src_layer_spec '{
   "path":"'${CORGIE_WALKTHROUGH_PATH}'/mask/fold_mask",
      "type": "mask",
      "name": "fold_mask"
   }' \
   --src_layer_spec '{
      "path":"'${CORGIE_WALKTHROUGH_PATH}'/img/unaligned",
      "args": {"binarization": ["eq", 0.0]},
      "type": "mask",
      "name": "black_mask"
   }' \
--dst_folder ${CORGIE_WALKTHROUGH_PATH} \
--stats_mip 7 --mip_start 6 --mip_end 8 \
--start_coord "150000, 150000, 17000" \
--end_coord "200000, 200000, 17010" \
--chunk_xy 1024 \
--suffix norm

Documentation

corgie normalize

Normalize image stats per a section (z-slice).

In many cases alignment procedures perform better when each input section has 0.0 mean and 1.0 variance. Statistics are collected from the whole section as as normalizing each chunk independently can produce significant border artifacts.

To exclude non-tissue pixels such image defects and plastic this command takes in an arbitrary number of mask layers as input.

corgie normalize [OPTIONS]

Options

-s, --src_layer_spec <src_layer_spec>

Required Source layer spec. Use multiple times to include all masks, fields, images. [required]

Format: JSON string.

Optional keys:

“type”: str from [‘img’, ‘segmentation’, ‘field’, ‘mask’, ‘section_value’, ‘fixed_field’, ‘float_tensor’] ;

“name”: str, DEFAULT – same as layer type. Required if given more than 1 layer of the same type ;

“data_backend”: str from [‘cv’, ‘json’] ;

“args”: a dictionary with additional layer parameters,

eg binarization scheme for masks data_mip_ranges, etc;

Required keys: “path”.

--dst_folder <dst_folder>: Required Folder where aligned stack will go

-s, --suffix <suffix>

--recompute_stats, --no_recompute_stats

Default: True

-m, --stats_mip <stats_mip>

-m, --mip_start <mip_start>: Required

-e, --mip_end <mip_end>: Required

-c, --chunk_xy <chunk_xy>

Default: 2048

--chunk_z <chunk_z>

Default: 1

--force_chunk_xy <force_chunk_xy>

--mask_value <mask_value>

Default: 0.0

--start_coord <start_coord>: Required

--end_coord <end_coord>: Required

--coord_mip <coord_mip>

Default: 0

-q, --queue_name <queue_name>: Name of the queue where the tasks will be pushed to. For file queue, use ‘fq://{path_to_shared_storage}’ format.If no ‘fq://’ prefix is given, the queue is assumed to be SQS. If not specified, tasks will be executed locally.

--completion_queue_name <completion_queue_name>: Name of AWS SQS queue where completion of tasks will be reported. Must be distinct from the ‘queue_name’. Providing a completion queue will speed up execution when running workers on unreliable machines (preemptible, spot).

--sqs_queue_region <sqs_queue_region>: AWS region of SQS queues. Task queue and completion queue must share the same region.

--restart_from_checkpoint_file <restart_from_checkpoint_file>: Option to be used in case of a job interruption. Specify path to filename where job progress is logged so that execution can start from where it left off before the interruption.

-d, --device <device>

Pytorch device specification. Eg: ‘cpu’, ‘cuda’, ‘cuda:0’

Default: cpu

-v, --verbose: Turn on debug logging

align-block

Example

corgie align-block \
--src_layer_spec '{"path": "'${CORGIE_WALKTHROUGH_PATH}'/img/img_norm"}' \
--dst_folder ${CORGIE_WALKTHROUGH_PATH}/aligned_blockmatch \
--start_coord "100000, 100000, 17000" \
--end_coord "150000, 150000, 17010" \
--chunk_xy 2048 \
--render_chunk_xy 2048 \
--suffix run_x0 \
--processor_spec '{"ApplyModel": {
   "params": {
      "path": "https://storage.googleapis.com/corgie_package/models/aligners/blockmatch",
      "tile_size": 128,
      "tile_step": 64,
      "max_disp": 48,
      "r_delta": 1.3
  }}}' \
--processor_mip 7

Documentation

corgie align-block

Align a sequential range of sections (a block).

corgie align-block [OPTIONS]

Options

-s, --src_layer_spec <src_layer_spec>

Required Source layer spec. Use multiple times to include all masks, fields, images. [required]

Format: JSON string.

Optional keys:

“type”: str from [‘img’, ‘segmentation’, ‘field’, ‘mask’, ‘section_value’, ‘fixed_field’, ‘float_tensor’] ;

“name”: str, DEFAULT – same as layer type. Required if given more than 1 layer of the same type ;

“data_backend”: str from [‘cv’, ‘json’] ;

“args”: a dictionary with additional layer parameters,

eg binarization scheme for masks data_mip_ranges, etc;

Required keys: “path”.

--dst_folder <dst_folder>: Required Folder where aligned stack will go

--suffix <suffix>

--seethrough_spec <seethrough_spec>: Seethrough method spec. Use multiple times to specify different methods (e.g. seethrough misalignments, seethrough black, etc.)

--seethrough_limit <seethrough_limit>: For each seethrough method, how many sections are allowed to be seenthrough. 0 or None means no limit.

--seethrough_spec_mip <seethrough_spec_mip>

--render_pad <render_pad>

Default: 512

--render_chunk_xy <render_chunk_xy>

Default: 1024

--processor_spec <processor_spec>: Required

-c, --chunk_xy <chunk_xy>

Default: 1024

--blend_xy <blend_xy>

Default: 0

--force_chunk_xy <force_chunk_xy>

--pad <pad>

Default: 64

--crop <crop>

-m, --processor_mip <processor_mip>: Required

--copy_start, --no_copy_start

Default: True

--use_starters, --no_starters

Default: True

--mode <mode>

Default: forward
Options: forward | backward | bidirectional

--vote_dist <vote_dist>

Default: 1

--start_coord <start_coord>: Required

--end_coord <end_coord>: Required

--coord_mip <coord_mip>

Default: 0

-q, --queue_name <queue_name>: Name of the queue where the tasks will be pushed to. For file queue, use ‘fq://{path_to_shared_storage}’ format.If no ‘fq://’ prefix is given, the queue is assumed to be SQS. If not specified, tasks will be executed locally.

--completion_queue_name <completion_queue_name>: Name of AWS SQS queue where completion of tasks will be reported. Must be distinct from the ‘queue_name’. Providing a completion queue will speed up execution when running workers on unreliable machines (preemptible, spot).

--sqs_queue_region <sqs_queue_region>: AWS region of SQS queues. Task queue and completion queue must share the same region.

--restart_from_checkpoint_file <restart_from_checkpoint_file>: Option to be used in case of a job interruption. Specify path to filename where job progress is logged so that execution can start from where it left off before the interruption.

-d, --device <device>

Pytorch device specification. Eg: ‘cpu’, ‘cuda’, ‘cuda:0’

Default: cpu

-v, --verbose: Turn on debug logging

align

Documentation

corgie align

Partition blocks and run align-block on each.

Blocks are merged together using decayed transforms.

corgie align [OPTIONS]

Options

-s, --src_layer_spec <src_layer_spec>

Required Source layer spec. Use multiple times to include all masks, fields, images. [required]

Format: JSON string.

Optional keys:

“type”: str from [‘img’, ‘segmentation’, ‘field’, ‘mask’, ‘section_value’, ‘fixed_field’, ‘float_tensor’] ;

“name”: str, DEFAULT – same as layer type. Required if given more than 1 layer of the same type ;

“data_backend”: str from [‘cv’, ‘json’] ;

“args”: a dictionary with additional layer parameters,

eg binarization scheme for masks data_mip_ranges, etc;

Required keys: “path”.

--dst_folder <dst_folder>: Required Folder where aligned stack will go

--suffix <suffix>

--align_blocks, --no_align_blocks

Default: True

--compute_stitch_fields, --no_compute_stitch_fields

Default: True

--compose_fields, --no_compose_fields

Default: True

--final_render, --no_final_render

Default: True

--render_pad <render_pad>

Default: 512

--render_chunk_xy <render_chunk_xy>

Default: 1024

--processor_spec <processor_spec>: Required

-m, --processor_mip <processor_mip>: Required

-c, --chunk_xy <chunk_xy>

Default: 1024

--blend_xy <blend_xy>

Default: 0

--force_chunk_xy <force_chunk_xy>

--pad <pad>

Default: 64

--crop <crop>

--seethrough_spec <seethrough_spec>: Seethrough method spec. Use multiple times to specify different methods (e.g. seethrough misalignments, seethrough black, etc.)

--seethrough_limit <seethrough_limit>: For each seethrough method, how many sections are allowed to be seenthrough. 0 or None means no limit.

--seethrough_spec_mip <seethrough_spec_mip>

--consensus_threshold <consensus_threshold>

A set of vectors can be considered “in consensus” if their consensus score is within the CONSENSUS_THRESHOLD of the lowest scoring set. The vector with the highest priority (the vector generated by the nearest section) that is considered “in consensus” will be considered the consensus vector for that location.

Default: 3.0

--blur_sigma <blur_sigma>

Determining the consensus vectors for all locations can be considered a one-hot weighting. This will introduce artificial non-smoothness in the final displacement field as vectors generated by one section transition to those generated by another section. This is adjusted by blurring the weightings so that the transitions are smoothed. BLUR_SIGMA controls the standard deviation of the Gaussian kernel used to blur. For more information see https://github.com/seung-lab/torchfields.

Default: 15.0

--kernel_size <kernel_size>

KERNEL_SIZE controls the width of the blurring kernel described in BLUR_SIGMA.

Default: 32

--start_coord <start_coord>: Required

--end_coord <end_coord>: Required

--coord_mip <coord_mip>

Default: 0

--bad_starter_path <bad_starter_path>: Filepath to list of sections that should not be used as the first section of any block.

--block_size <block_size>

Default: 10

--stitch_size <stitch_size>

The number of sections of overlap that will be used to generate stitch fields. If STITCH_SIZE > 1, there will be multiple estimates of the stitch field, and the final stitch field will be voted for consensus.

Default: 3

--vote_dist <vote_dist>

The number of previously aligned sections to use as target sections during block alignment. If VOTE_DIST > 1, then there will be multiple estimates of the block field, and the final block field will be voted for consensus.

Default: 1

--decay_dist <decay_dist>

The maximum distance (measured in sections) for which a previous starter section may influence the final displacement field of a later section after stitching. Currently, the stitch field for a starter section is attenuated before being composed with a section’s block field to produce the final displacement field. This attenuation is a single coefficient, min(max(1.0 - (CurrentSection - StarterSection) / DECAY_DIST, 0.0), 1.0).

Default: 100

--blur_rate <blur_rate>

The rate of increase in the width of the blurring kernel used during stitching as distance of a section from a previous starter section increases. Can be roughly considered the change in size of the standard deviation of a Gaussian kernel per single section (we currently create the downsample hierarchy with a box filter). For example, if the standard deviation should increase by 1 px every 10 sections, then set BLUR_RATE = 0.1.

Default: 0.2

--restart_stage <restart_stage>

0: align blocks

1: align overlapping regions of neighboring blocks

2: vote over overlaps to produce stitch fields

3: compose stitch fields with weight decay to produce final displacement fields

4: render the image with the final field at the resolution of the final field

Default: 0

--restart_suffix <restart_suffix>: The suffix of a previous alignment run, which will be used as a starting point. New layers created after the restart will be labeld with –suffix

-q, --queue_name <queue_name>: Name of the queue where the tasks will be pushed to. For file queue, use ‘fq://{path_to_shared_storage}’ format.If no ‘fq://’ prefix is given, the queue is assumed to be SQS. If not specified, tasks will be executed locally.

--completion_queue_name <completion_queue_name>: Name of AWS SQS queue where completion of tasks will be reported. Must be distinct from the ‘queue_name’. Providing a completion queue will speed up execution when running workers on unreliable machines (preemptible, spot).

--sqs_queue_region <sqs_queue_region>: AWS region of SQS queues. Task queue and completion queue must share the same region.

--restart_from_checkpoint_file <restart_from_checkpoint_file>: Option to be used in case of a job interruption. Specify path to filename where job progress is logged so that execution can start from where it left off before the interruption.

-d, --device <device>

Pytorch device specification. Eg: ‘cpu’, ‘cuda’, ‘cuda:0’

Default: cpu

-v, --verbose: Turn on debug logging