birl.utilities.evaluate module¶
Evaluate experiments
Copyright (C) 2016-2019 Jiri Borovec <jiri.borovec@fel.cvut.cz>
-
birl.utilities.evaluate.
aggregate_user_score_timeline
(df, col_aggreg, col_user, col_score, lower_better=True, top_down=True, interp=False)[source]¶ compute some cumulative statistic over given table, assuming col_aggreg is continues first it is grouped by col_aggreg and chose min/max (according to lower_better) of col_score assuming that col_aggreg is sortable like a timeline do propagation of min/max from past values depending on top_down (which reverse the order)
Parameters: - df – rich table containing col_aggreg, col_user, col_score
- col_aggreg (str) – used for grouping assuming to be like a timeline
- col_user (str) – by this column the scores are assumed to be independent
- col_score (str) – the scoring value for selecting the best
- lower_better (bool) – taking min/max of scoring value
- top_down (bool) – reversing the order according to col_aggreg
- interp (bool) – in case some scores for col_aggreg are missing, interpolate from past
Return DF: table
>>> np.random.seed(0) >>> df = pd.DataFrame() >>> df['day'] = np.random.randint(0, 5, 50) >>> df['user'] = np.array(list('abc'))[np.random.randint(0, 3, 50)] >>> df['score'] = np.random.random(50) >>> df_agg = aggregate_user_score_timeline(df, 'day', 'user', 'score') >>> df_agg.round(3) # doctest: +NORMALIZE_WHITESPACE b c a 4 0.447 0.132 0.567 0 0.223 0.005 0.094 3 0.119 0.005 0.094 1 0.119 0.005 0.094 2 0.119 0.005 0.020
-
birl.utilities.evaluate.
compute_affine_transf_diff
(points_ref, points_init, points_est)[source]¶ compute differences between initial state and estimated results
Parameters: - points_ref (ndarray) – np.array<nb_points, dim>
- points_init (ndarray) – np.array<nb_points, dim>
- points_est (ndarray) – np.array<nb_points, dim>
Return ndarray: list of errors
>>> points_ref = np.array([[1, 2], [3, 4], [2, 1]]) >>> points_init = np.array([[3, 4], [1, 2], [2, 1]]) >>> points_est = np.array([[3, 4], [2, 1], [1, 2]]) >>> diff = compute_affine_transf_diff(points_ref, points_init, points_est) >>> import pandas as pd >>> pd.Series(diff).sort_index() # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Affine rotation Diff -8.97... Affine scale X Diff -0.08... Affine scale Y Diff -0.20... Affine shear Diff -1.09... Affine translation X Diff -1.25... Affine translation Y Diff 1.25... dtype: float64
>>> # Wrong input: >>> compute_affine_transf_diff(None, np.array([[1, 2], [3, 4], [2, 1]]), None) {}
-
birl.utilities.evaluate.
compute_matrix_user_ranking
(df_stat, higher_better=False)[source]¶ compute ranking matrix over features in columns sorting per column and unique colour per user
Parameters: - df_stat (DF) – table where index are users and columns are scoring
- higher_better (bool) – ranking such that larger value is better
Return ndarray: ranking with features in columns
>>> np.random.seed(0) >>> df = pd.DataFrame(np.random.random((5, 3)), columns=list('abc')) >>> compute_matrix_user_ranking(df) # doctest: +NORMALIZE_WHITESPACE array([[ 3., 1., 4.], [ 2., 0., 3.], [ 1., 3., 0.], [ 0., 2., 1.], [ 4., 4., 2.]])
-
birl.utilities.evaluate.
compute_ranking
(user_cases, field, reverse=False)[source]¶ compute ranking over selected field
Parameters: Return dict(dict(dict)): extended dictionary
>>> user_cases = { ... 'karel': {1: {'rTRE': 0.04}, 2: {'rTRE': 0.25}, 3: {'rTRE': 0.1}}, ... 'pepa': {1: {'rTRE': 0.33}, 3: {'rTRE': 0.05}}, ... 'franta': {2: {'rTRE': 0.01}, 3: {'rTRE': 0.15}} ... } >>> user_cases = compute_ranking(user_cases, 'rTRE') >>> import pandas as pd >>> df = pd.DataFrame({usr: {cs: user_cases[usr][cs]['rTRE_rank'] ... for cs in user_cases[usr]} ... for usr in user_cases})[sorted(user_cases.keys())] >>> df # doctest: +NORMALIZE_WHITESPACE franta karel pepa 1 3 1 2 2 1 2 3 3 3 2 1
-
birl.utilities.evaluate.
compute_target_regist_error_statistic
(points_ref, points_est)[source]¶ compute distance as between related points in two sets and make a statistic on those distances - mean, std, median, min, max
Parameters: - points_ref (ndarray) – final landmarks in target image of np.array<nb_points, dim>
- points_est (ndarray) – warped landmarks from source to target of np.array<nb_points, dim>
Return tuple(ndarray,dict): (np.array<nb_points, 1>, dict)
>>> points_ref = np.array([[1, 2], [3, 4], [2, 1]]) >>> points_est = np.array([[3, 4], [2, 1], [1, 2]]) >>> dist, stat = compute_target_regist_error_statistic(points_ref, points_ref) >>> dist array([ 0., 0., 0.]) >>> all(stat[k] == 0 for k in stat if k not in ['overlap points']) True >>> dist, stat = compute_target_regist_error_statistic(points_ref, points_est) >>> dist #doctest: +ELLIPSIS array([ 2.828..., 3.162..., 1.414...]) >>> import pandas as pd >>> pd.Series(stat).sort_index() # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Max 3.16... Mean 2.46... Mean_weighted 2.52... Median 2.82... Min 1.41... STD 0.75... overlap points 1.00... dtype: float64
>>> # Wrong input: >>> compute_target_regist_error_statistic(None, np.array([[1, 2], [3, 4], [2, 1]])) ([], {'overlap points': 0})
-
birl.utilities.evaluate.
compute_tre
(points_1, points_2)[source]¶ computing Target Registration Error for each landmark pair
Parameters: - points_1 (ndarray) – set of points
- points_2 (ndarray) – set of points
Return ndarray: list of errors of size min nb of points
>>> np.random.seed(0) >>> compute_tre(np.random.random((6, 2)), ... np.random.random((9, 2))) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE array([ 0.21..., 0.70..., 0.44..., 0.34..., 0.41..., 0.41...])
-
birl.utilities.evaluate.
compute_tre_robustness
(points_target, points_init, points_warp)[source]¶ compute robustness as improvement for each TRE
Parameters: - points_target (ndarray) – final landmarks in target image
- points_init (ndarray) – initial landmarks in source image
- points_warp (ndarray) – warped landmarks from source to target
Return bool: improvement
>>> np.random.seed(0) >>> compute_tre_robustness(np.random.random((10, 2)), ... np.random.random((9, 2)), ... np.random.random((8, 2))) 0.375 >>> compute_tre_robustness(np.random.random((10, 2)), ... np.random.random((9, 2)) + 5, ... np.random.random((8, 2)) + 2) 1.0
-
birl.utilities.evaluate.
grouping_cumulative
(df, col_index, col_column)[source]¶ compute histogram statistic over selected column and in addition group this histograms
Parameters: Return DF: >>> np.random.seed(0) >>> df = pd.DataFrame() >>> df['result'] = np.random.randint(0, 2, 50) >>> df['user'] = np.array(list('abc'))[np.random.randint(0, 3, 50)] >>> grouping_cumulative(df, 'user', 'result').astype(int) # doctest: +NORMALIZE_WHITESPACE 0 1 user a 10 12 b 4 9 c 6 9