birl.benchmark module

General benchmark template for all registration methods. It also serves for evaluating the input registration pairs (while no registration is performed, there is only the initial deformation)

Sample run (usage):

mkdir ./results
python benchmarks/bm_registration.py         -t data-images/pairs-imgs-lnds_histol.csv -d ./data-images         -o ./results --unique

Copyright (C) 2016-2019 Jiri Borovec <jiri.borovec@fel.cvut.cz>

class birl.benchmark.ImRegBenchmark(params)

Bases: birl.utilities.experiments.Experiment

General benchmark class for all registration methods. It also serves for evaluating the input registration pairs.

Parameters

params (dict) – dictionary with experiment configuration, the required options are names in REQUIRED_PARAMS, note that the basic parameters are inherited

The benchmark has following steps:

1. check all necessary paths and required parameters

2. load cover file and set all paths as absolute

3. run individual registration experiment in sequence or in parallel

   (nb_workers > 1); if the particular experiment folder exist (assume completed experiment) and skip it:

   1. create experiment folder and init experiment

   2. generate execution command

   3. run the command (an option to lock it in single thread)

   4. evaluate experiment, set the expected outputs and visualisation

   5. clean all extra files if any

4. visualise results abd evaluate registration results

Note

The actual implementation simulates the “IDEAL” registration while it blindly copies the reference landmarks as results of the registration. In contrast to the right registration, it copies the moving images so there is alignment (consistent warping) between resulting landmarks and image.

Examples

>>> # Running in single thread:
>>> from birl.utilities.data_io import create_folder, update_path
>>> path_out = create_folder('temp_results')
>>> path_csv = os.path.join(update_path('data-images'), 'pairs-imgs-lnds_mix.csv')
>>> params = {'path_table': path_csv,
...           'path_out': path_out,
...           'nb_workers': 1,
...           'unique': False,
...           'visual': True}
>>> benchmark = ImRegBenchmark(params)
>>> benchmark.run()
True
>>> shutil.rmtree(path_out, ignore_errors=True)

>>> # Running in multiple parallel threads:
>>> from birl.utilities.data_io import create_folder, update_path
>>> path_out = create_folder('temp_results')
>>> path_csv = os.path.join(update_path('data-images'), 'pairs-imgs-lnds_mix.csv')
>>> params = {'path_table': path_csv,
...           'path_out': path_out,
...           'nb_workers': 2,
...           'unique': False,
...           'visual': True}
>>> benchmark = ImRegBenchmark(params)
>>> benchmark.run()
True
>>> shutil.rmtree(path_out, ignore_errors=True)

initialise benchmark

Parameters

params (dict) – parameters

__check_exist_regist(idx, path_dir_reg)

check whether the particular experiment already exists and have results

if the folder with experiment already exist and it is also part of the loaded finished experiments, sometimes the oder may mean failed experiment

Parameters

• idx (int) – index of particular

• path_dir_reg (str) –

Return bool

__execute_method(method, input_table, path_csv=None, desc='', aggr_experiments=False, nb_workers=None)

execute a method in sequence or parallel

Parameters

• method (func) – used method

• input_table (DF) – iterate over table

• path_csv (str) – path to the output temporal csv

• desc (str) – name of the running process

• aggr_experiments (bool) – append output to experiment DF

• nb_workers (int|None) – number of jobs, by default using class setting

Returns

__export_df_experiments(path_csv=None)

export the DataFrame with registration results

Parameters

| None path_csv (str) – path to output CSV file

__images_preprocessing(item)

create some pre-process images, convert to gray scale and histogram matching

Parameters

item (dict) – the input record

Return dict

updated item with optionally added pre-process images

__remove_pproc_images(item)

remove preprocess (temporary) image if they are not also final

Parameters

item (dict) – the input record

Return dict

updated item with optionally removed temp images

_absolute_path(path, destination='data', base_path='')

update te path to the dataset or output

Parameters

• path (str) – original path

• destination (str) – type of update data for data source and expt for output experimental folder

• destination – type of update

Return str

updated path

classmethod _clear_after_registration(item)

clean unnecessarily files after the registration

Parameters

item (dict) – dictionary with regist. information

Return dict

the same or updated regist. info

_copy_config_to_expt(field_path)

copy particular configuration to the experiment folder

Parameters

field_path (str) – field from parameters containing a path to file

_evaluate()

evaluate complete benchmark experiment

_execute_img_registration(item)

execute the image registration itself

Parameters

item (dict) – record

Return dict

record

_extract_execution_time(item)

if needed update the execution time

Parameters

item (dict) – dictionary {str: value} with registration params

Return float|None

time in minutes

classmethod _extract_warped_image_landmarks(item)

get registration results - warped registered images and landmarks

Parameters

item (dict) – dictionary with registration params

Return dict

paths to warped images/landmarks

_generate_regist_command(item)

generate the registration command(s)

Parameters

item (dict) – dictionary with registration params

Return str|list(str)

the execution commands

_get_path_reg_dir(item)

_get_paths(item, prefer_pproc=True)

expand the relative paths to absolute, if TEMP path is used, take it

Parameters

• item (dict) – row from cover file with relative paths

• prefer_pproc (bool) – prefer using preprocess images

Return tuple(str,str,str,str)

path to reference and moving image and reference and moving landmarks

classmethod _image_diag(item, path_img_ref=None)

get the image diagonal from several sources

1. diagonal exists in the table

2. image size exist in the table

3. reference image exists

Parameters

• item (dict|DF) – one row from the table

• path_img_ref (str) – optional path to the reference image

Return float|None

image diagonal

_load_data()

loading data, the cover file with all registration pairs

classmethod _load_landmarks(item, path_dataset)

classmethod _load_warped_image(item, path_experiment=None)

load the wapted image if it exists

Parameters

• item (dict) – row with the experiment

• path_experiment (str|None) – path to the experiment folder

Return ndarray

_parse_regist_results(item)

evaluate rests of the experiment and identity the registered image and landmarks when the process finished

Parameters

item (dict) – dictionary {str: value} with registration params

Return dict

_perform_registration(df_row)

run single registration experiment with all sub-stages

Parameters

df_row (tuple(int,dict)) – row from iterated table

classmethod _prepare_img_registration(item)

prepare the experiment folder if it is required, eq. copy some extra files

Parameters

item (dict) – dictionary with regist. params

Return dict

the same or updated registration info

_relativize_path(path, destination='path_exp')

extract relative path according given parameter

Parameters

• path (str) – the original path to file/folder

• destination (str) – use path from parameters

Return str

relative or the original path

_run()

perform complete benchmark experiment

_summarise()

summarise benchmark experiment

classmethod _visual_image_move_warp_lnds_move_warp(item, path_dataset=None, path_experiment=None)

visualise the case with warped moving image and landmarks to the reference frame so they are simple to overlap

Parameters

• item (dict) – row with the experiment

• path_dataset (str|None) – path to the dataset folder

• path_experiment (str|None) – path to the experiment folder

Return obj|None

classmethod _visual_image_move_warp_lnds_ref_warp(item, path_dataset=None, path_experiment=None)

visualise the case with warped reference landmarks to the move frame

Parameters

• item (dict) – row with the experiment

• path_dataset (str|None) – path to the dataset folder

• path_experiment (str|None) – path to the experiment folder

Return obj|None

classmethod compute_registration_accuracy(df_experiments, idx, points1, points2, state='', img_diag=None, wo_affine=False)

compute statistic on two points sets

IRE - Initial Registration Error TRE - Target Registration Error

Parameters

• df_experiments (DF) – DataFrame with experiments

• idx (int) – index of tha particular record

• points1 (ndarray) – np.array<nb_points, dim>

• points2 (ndarray) – np.array<nb_points, dim>

• state (str) – whether it was before of after registration

• img_diag (float) – target image diagonal

• wo_affine (bool) – without affine transform, assume only local/elastic deformation

classmethod compute_registration_statistic(idx_row, df_experiments, path_dataset=None, path_experiment=None, path_reference=None)

after successful registration load initial nad estimated landmarks afterwords compute various statistic for init, and final alignment

Parameters

• df_row (tuple(int,dict)) – row from iterated table

• df_experiments (DF) – DataFrame with experiments

• path_dataset (str|None) – path to the provided dataset folder

• path_reference (str|None) – path to the complete landmark collection folder

• path_experiment (str|None) – path to the experiment folder

static extend_parse(arg_parser)

classmethod main(params=None)

run the Main of selected experiment

Parameters

• cls – class of selected benchmark

• params (dict) – set of input parameters

classmethod visualise_registration(idx_row, path_dataset=None, path_experiment=None)

visualise the registration results according what landmarks were estimated - in registration or moving frame

Parameters

• df_row (tuple(int,dict)) – row from iterated table

• path_dataset (str) – path to the dataset folder

• path_experiment (str) – path to the experiment folder

COL_IMAGE_DIAGONAL = 'Image diagonal [pixels]'

image diagonal in pixels

COL_IMAGE_EXT_TEMP = ' TEMP'

extension to the image column name for temporary pre-process image

COL_IMAGE_MOVE = 'Source image'

moving (registration source) image

COL_IMAGE_MOVE_WARP = 'Warped source image'

moving image warped to the reference frame

COL_IMAGE_REF = 'Target image'

reference (registration target) image

COL_IMAGE_REF_WARP = 'Warped target image'

reference image warped to the moving frame

COL_IMAGE_SIZE = 'Image size [pixels]'

tuple of image size

COL_NB_LANDMARKS_INPUT = 'nb. dataset landmarks'

number of landmarks in dataset (min of moving and reference)

COL_NB_LANDMARKS_WARP = 'nb. warped landmarks'

number of warped landmarks

COL_POINTS_MOVE = 'Source landmarks'

moving (registration source) landmarks

COL_POINTS_MOVE_WARP = 'Warped source landmarks'

moving landmarks warped to the reference frame

COL_POINTS_REF = 'Target landmarks'

reference (registration target) landmarks

COL_POINTS_REF_WARP = 'Warped target landmarks'

reference landmarks warped to the moving frame

COL_REG_DIR = 'Registration folder'

registration folder for each particular experiment

COL_ROBUSTNESS = 'Robustness'

define robustness as improved image alignment from initial state

COL_STATUS = 'status'

define train / test status

COL_TIME = 'Execution time [minutes]'

measured time of image registration in minutes

COL_TIME_PREPROC = 'Pre-processing time [minutes]'

measured time of image pre-processing in minutes

COVER_COLUMNS = ('Target image', 'Source image', 'Target landmarks', 'Source landmarks')

COVER_COLUMNS_EXT = ('Target image', 'Source image', 'Target landmarks', 'Source landmarks', 'Image size [pixels]', 'Image diagonal [pixels]')

COVER_COLUMNS_WRAP = ('Target image', 'Source image', 'Target landmarks', 'Source landmarks', 'Warped target image', 'Warped source image', 'Warped target landmarks', 'Warped source landmarks')

EXECUTE_TIMEOUT = 3600

does not work for Py2

Type

timeout for executing single image registration, NOTE

NAME_CSV_REGISTRATION_PAIRS = 'registration-results.csv'

some needed files

NAME_IMAGE_MOVE_WARP_POINTS = 'image_warped_landmarks_warped.jpg'

output image name in experiment folder for reg. results - image and landmarks are warped

NAME_IMAGE_REF_POINTS_WARP = 'image_ref_landmarks_warped.jpg'

output image name in experiment folder for reg. results - warped landmarks in reference image

NAME_IMAGE_REF_WARP = 'image_refence-warped.jpg'

output image name in experiment folder for reg. results - overlap of reference and warped image

NAME_IMAGE_WARPED_VISUAL = 'registration_visual_landmarks.jpg'

output image name in experiment folder for showing improved alignment by used registration

NAME_LOG_REGISTRATION = 'registration.log'

logging file for registration experiments

NAME_RESULTS_CSV = 'results-summary.csv'

default file for exporting results in table format

NAME_RESULTS_TXT = 'results-summary.txt'

default file for exporting results in formatted text format

NB_WORKERS_USED = 1

default number of threads used by benchmarks

REQUIRED_PARAMS = ['path_out', 'path_table']

required experiment parameters

birl.benchmark._df_drop_unnamed(df)

Drop columns was index without name and was loaded as Unnamed: 0.

birl.benchmark.export_summary_results(df_experiments, path_out, params=None, name_txt='results-summary.txt', name_csv='results-summary.csv')

export the summary as CSV and TXT

Parameters

• df_experiments (DF) – DataFrame with experiments

• path_out (str) – path to the output folder

• params (dict|None) – experiment parameters

• name_csv (str) – results file name

• name_txt (str) – results file name

>>> export_summary_results(pd.DataFrame(), '')

birl.benchmark.filter_paired_landmarks(item, path_dataset, path_reference, col_source, col_target)

filter all relevant landmarks which were used and copy them to experiment

The case is that in certain challenge stage users had provided just a subset

of all image landmarks which could be laos shuffled. The idea is to filter identify all user used (provided in dataset) landmarks and filter them from temporary reference dataset.

Parameters

• item (dict|Series) – experiment DataFrame

• path_dataset (str) – path to provided landmarks

• path_reference (str) – path to the complete landmark collection

• col_source (str) – column name of landmarks to be transformed

• col_target (str) – column name of landmarks to be compared

Return tuple(float,ndarray,ndarray)

match ratio, filtered ref and move landmarks

>>> p_data = update_path('data-images')
>>> p_csv = os.path.join(p_data, 'pairs-imgs-lnds_histol.csv')
>>> df = pd.read_csv(p_csv)
>>> ratio, lnds_ref, lnds_move = filter_paired_landmarks(dict(df.iloc[0]), p_data, p_data,
...     ImRegBenchmark.COL_POINTS_MOVE, ImRegBenchmark.COL_POINTS_REF)
>>> ratio
1.0
>>> lnds_ref.shape == lnds_move.shape
True

birl.benchmark.COL_PAIRED_LANDMARKS = 'Ration matched landmarks'

In case provided dataset and complete (true) dataset differ