API documentation¶
Main routine¶
- thetis.thetis(thetis_args: ThetisArgs) dict¶
Main entrance point for the Thetis evaluation service.
Executes the main Thetis processing function after initializing the license and validating the user configuration. Handles exceptions and wraps the core logic in a license context to ensure compliance.
Given a ground-truth data set and the respective predictions by an AI model, this service examines several safety-relevant aspects such as uncertainty quality, data set quality, performance metrics, and fairness.
- Parameters:
thetis_args (ThetisArgs) – A data class containing all necessary arguments for the execution, including the API key, license file path, license XML string, and license key/signature.
- Returns:
- Dictionary with the evaluation results, rating scores, and
recommendations for the examined AI model.
- Return type:
dict
- Raises:
ConnectionError – if API key is given but if connection to Thetis user management cannot be established. (raised by
thetiscore._licensing.license._License.__init__()).LicenseInvalidError – if API key is given but if API key is either invalid or a valid license cannot be found. (raised by
thetiscore._licensing.license._License.__init__()).ThetisInternalLicenseError – on any other unexpected error during API key validation. (raised by
thetiscore._licensing.license._License.__init__()).FileNotFoundError – if file with given license_file_path cannot be found (raised by
thetiscore._licensing.license._License.__init__()).RuntimeError – if license_file_path is given but file path is malformed (e.g., due to insufficient escaping of backslashes) (raised by
thetiscore._licensing.license._License.__init__()).RuntimeError – if parsing the given XML file failed (raised by
thetiscore._licensing.license._License.__init__()).RuntimeError – if parsing the given XML string failed (raised by
thetiscore._licensing.License.__init__()).RuntimeError – if neither ‘license_file_path’, ‘license_xml_str’, ‘license_key_and_signature’, THETIS_LICENSE, “license.dat” in working directory nor “license.dat” in user local home directory (OS specific) could be found (raised by
thetiscore._licensing.license._License.__init__()).AttributeError – if the key ‘Key’ cannot be found at the expected location (raised by
thetiscore._licensing.license._License.__init__()).AttributeError – if the key ‘Signature’ cannot be found at the expected location (raised by
thetiscore._licensing.license._License.__init__()).thetiscore.errors.LicenseInvalidError – if the passed license key/signature pair is invalid (raised by
thetiscore._licensing.license._License.verify()).RuntimeError – if the license Key XML cannot be parsed (raised by
thetiscore._licensing.license._License.verify()).AttributeError – if the key ‘ExpiryDate’ cannot be found at the expected location (raised by
thetiscore._licensing.license._License.verify()).thetiscore.errors.LicenseExpiredError – if the license has expired (raised by
thetiscore._licensing.license._License.verify()).TypeError – if config is neither str nor python dict (raised by
thetiscore._data.get_config()).SyntaxError – if the syntax of the user configuration is malformed according to the required configuration schema (raised by
thetiscore._data.get_config()).AttributeError – if classification or detection mode but key “distinct_classes” is missing or empty (raised by
thetiscore._data.get_config()).AttributeError – if classification mode but length of “distinct_classes” is lower than 2 (raised by
thetiscore._data.get_config()).AttributeError – if binary classification mode (length of “distinct_classes” is 2) but error_msg field “binary_positive_label” is missing (raised by
thetiscore._data.get_config()).AttributeError – if detection mode but key “task_settings/detection_bbox_format” is missing in user config. (raised by
thetiscore._data.get_config()).AttributeError – if “fairness” evaluation is active but no sensitive attribute has been specified (raised by
thetiscore._data.get_config()).AttributeError – if more than 2 distinct_classes are specified along with a “binary_positive_label” (raised by
thetiscore._data.get_config()).AttributeError – if “binary_positive_label” key is specified for any task other than classification (raised by
thetiscore._data.get_config()).AttributeError – if “distinct_classes” key is specified for a regression task (raised by
thetiscore._data.get_config()).RuntimeError – if the license Key XML cannot be parsed (raised by
thetiscore._licensing.license._License.verify_feature()).AttributeError – if the key ‘Features’ cannot be found at the expected location (raised by
thetiscore._licensing.license._License.verify_feature()).NotImplementedError – if “task” is not one of “classification”, “regression”, “detection” (raised by
thetiscore._data.get_input()).RuntimeError – if the given language identifier is unknown or not supported (raised by
thetiscore._language.Language._Language.__init__()).TypeError – if ‘predictions’ is not type pd.DataFrame (classification, raised by
thetiscore._data.get_input()).TypeError – if ‘annotations’ is not type pd.DataFrame (classification, raised by
thetiscore._data.get_input()).TypeError – if ‘predictions’ is not type dict (detection, raised by
thetiscore._data.get_input()).TypeError – if any value in ‘predictions’ dictionary is not type pd.DataFrame (detection, raised by
thetiscore._data.get_input()).TypeError – if ‘annotations’ is not type dict (detection, raised by
thetiscore._data.get_input()).TypeError – if any value in ‘annotations’ dictionary is not type pd.DataFrame (detection, raised by
thetiscore._data.get_input()).TypeError – if ‘description’ is not type dict
TypeError – if ‘description’ values are not type str
KeyError – if ‘description’ has unexpected or missing keys
RuntimeError – if the indices of the predicted and ground-truth data set do not match to each other. (classification, raised by
thetiscore._data.get_input()).ValueError – if “distinct_classes” in application config contains duplicates (classification or detection, raised by
thetiscore._data.get_input()).AttributeError – if the requested column “labels” does not exist within the predicted data set (classification or detection, raised by
thetiscore._data.get_input()).AttributeError – if the requested column “predictions” does not exist within the predicted data set (regression, raised by
thetiscore._data.get_input()).AttributeError – if the requested column “target” does not exist within the ground-truth data set (raised by
thetiscore._data.get_input()).ValueError – if the data type conversion to one of “int” or “str” of column “labels” failed within the predicted data set (classification or detection, raised by
thetiscore._data.get_input()).ValueError – if the data type conversion to “float” of column “predictions” failed within the predicted data set (regression, raised by
thetiscore._data.get_input()).ValueError – if the data type conversion to one of “int” or “str” of column “target” failed within the ground-truth data set (classification or detection, raised by
thetiscore._data.get_input()).ValueError – if the data type conversion to “float” of column “target” failed within the ground-truth data set (regression, raised by
thetiscore._data.get_input()).RuntimeError – if the arrays for predicted labels and ground-truth data set have different dtypes (classification or detection, raised by
thetiscore._data.get_input()).ValueError – if classes are found that have not been specified by ‘distinct_classes’ in the config file (classification or detection, raised by
thetiscore._data.get_input()).ValueError – if binary classification and “binary_positive_label” can not be found in “distinct_classes” (classification or detection, raised by
thetiscore._data.get_input()).ValueError – if binary classification and a complementary label to “binary_positive_label” can not be found in “distinct_classes” (classification or detection, raised by
thetiscore._data.get_input()).RuntimeError – if multi-class extraction failed and if more than 2 distinct classes are detected (binary classification, raised by
thetiscore._data.get_input()).AttributeError – if multi-class extraction failed and if the requested column “confidence” does not exist within the predicted data set (binary classification or detection, raised by
thetiscore._data.get_input()).ValueError – if multi-class extraction failed and if the data type conversion to “float” of column “confidence” failed within the predicted data set (binary classification or detection, raised by
thetiscore._data.get_input()).RuntimeError – if multi-class extraction failed and if the specified positive label within the application config cannot be found in the data set (binary classification, raised by
thetiscore._data.get_input()).AttributeError – if a sensitive feature is defined for a label that has not been found in the data set. (raised by
thetiscore._data.get_input()).RuntimeError – if uncertainty evaluation is active but no uncertainty information is given (regression, raised by
thetiscore._data.get_input()).ValueError – if column “variance” exists in predictions dataframe and if the data type conversion to “float” of column “variance” failed within the predicted data set (regression, raised by
thetiscore._data.get_input()).ValueError – if column “stddev” exists in predictions dataframe and if the data type conversion to “float” of column “stddev” failed within the predicted data set (regression, raised by
thetiscore._data.get_input()).AttributeError – if the requested column for a sensitive feature does not exist within the ground-truth data set (raised by
thetiscore._data.get_input()).ValueError – if the data type conversion to one of “int” or “str” failed for a requested sensitive feature within the ground-truth data set (raised by
thetiscore._data.get_input()).ValueError – if a sensitive feature has a missing or invalid entry (raised by
thetiscore._data.get_input()).ValueError – if a sensitive feature has less than 2 distinct labels (raised by
thetiscore._data.get_input()).RuntimeError – if bbox_format is not one of ‘xyxy’, ‘xywh’, or ‘cxcywh’ (detection, raised by:
thetiscore._data.get_input()).AttributeError – if the field ‘__meta__’ within the ground-truth data set annotations is completely missing (detection, raised by:
thetiscore._data.get_input()).AttributeError – if the meta information provided by ‘__meta__’ is missing for a certain image (detection, raised by:
thetiscore._data.get_input()).ValueError – if the image width or height information provided by ‘__meta__’ for a certain image are <= 0 (detection, raised by:
thetiscore._data.get_input()).ValueError – if bbox_format is ‘xyxy’ and xmin > xmax (detection, raised by:
thetiscore._data.get_input()).ValueError – if bbox_format is ‘xyxy’ and ymin > ymax (detection, raised by:
thetiscore._data.get_input()).ValueError – if bbox_format is ‘xywh’ or ‘cxcywh’ and width is negative (detection, raised by:
thetiscore._data.get_input()).ValueError – if bbox_format is ‘xywh’ or ‘cxcywh’ and height is negative (detection, raised by:
thetiscore._data.get_input()).RuntimeError – if the fields “weight” or “invert” are missing within a YAML file for a certain rating aspect (raised by
thetiscore._utils.get_rating_score()).RuntimeError – if “evaluation_spline” has been defined within a YAML file for a certain rating aspect but fieds “x” or “y” are missing. (raised by
thetiscore._utils.get_rating_score()).RuntimeError – if the “weight” parameters do not sum up to 1 during metric calculation. (raised by
thetiscore._utils.get_rating_score()).RuntimeError – if ‘rating_scores’ and ‘rating_aspects’ do not have the same length (raised by
thetiscore._utils.get_rating_recommendations()).ValueError – if the confidence for any prediction is greater than 1 (classification, raised by
thetiscore._data.get_input()).ValueError – if the confidence for any prediction is lower than 0 (classification, raised by
thetiscore._data.get_input()).ValueError – if there are any NaN or missing values in the required columns in either predictions or annotations (raised by
thetiscore._data.get_input()).NotImplementedError – if the bounding box matching strategy is not one of “exclusive”, “max” (raised by
thetiscore._data.get_input()).ValueError – if all prediction and annotation dataframes are empty, and task is set to object detection (raised by
thetiscore._data.get_input()).NotImplementedError – if detection_bbox_clipping is specified and is not one of ‘relative .%’ or ‘absolute .px’ (raised by
thetiscore._data.get_input()).RuntimeError – if detection_bbox_clipping value is not specified correctly (in case of relative or absolute px) (raised by
thetiscore._data.get_input()).RuntimeError – if task is detection and any boxes are out of bounds set by the detection_bbox_clipping parameter (raised by:
thetiscore._data.utils.get_input()).
MLflow format¶
- thetis.thetis_mlflow(thetist_args: ThetisArgs) dict¶
Entrance point for the Thetis evaluation service with MLFlow logging support.
Given a ground truth dataset and the respective predictions by an AI model, this function examines various aspects such as performance, data_quality, fairness, robustness, and uncertainty consistency.
It supports tasks like classification, detection, and regression.
Hint: you can also pass the current step count via ‘mlflow_step’ to this function. The step count will be passed to MLflow’s ‘log_metric’ function. Refer to the documentation for more details on individual metrics.
Note: You must provide one of the following arguments for license validation: license_file_path, license_xml_str, or license_key_and_signature.
- Metrics:
The set of metrics passed to MLflow depends on the selection of evaluation tasks. The following metrics are passed to MLflow when running in classification mode:
performance/accuracy- Classification accuracy.performance/balanced_accuracy- Balanced classification accuracy.performance/precision- Classification precision.performance/recall- Classification recall.performance/f1- Classification F1 score (harmonic mean of precision and recall).uncertainty/rating_score- Rating score of classification uncertainty quality/calibration.uncertainty/ece- Expected Calibration Error (ECE).uncertainty/mce- Maximum Calibration Error (MCE).uncertainty/nll- Negative Log Likelihood (NLL).uncertainty/brier- Brier Score (BS).fairness/rating_score- Rating score of classification fairness.fairness/demographic_parity_difference_score- Demographic parity difference.fairness/equalized_odds_difference_score- Equalized odds difference.data_evaluation/rating_score- Rating score of dataset quality.
The following metrics are passed to MLflow when running in detection mode:
performance/iou_<IoU score>/ap- Detection Average Precision for a certain IoU score.performance/iou_<IoU score>/precision- Detection precision for a certain IoU score.performance/iou_<IoU score>/recall- Detection recall for a certain IoU score.performance/iou_<IoU score>/f1- Detection F1 score (harmonic mean of precision and recall) for a certain IoU score.uncertainty/rating_score- Rating score of detection uncertainty quality/calibration.uncertainty/iou_<IoU score>/ece- Expected Calibration Error (ECE) for a certain IoU score.uncertainty/iou_<IoU score>/dece_xy- Detection Expected Calibration Error (D-ECE) which measures miscalibration w.r.t. cx/cy position for a certain IoU score.uncertainty/iou_<IoU score>/dece_wh- Detection Expected Calibration Error (D-ECE) which measures miscalibration width/height information for a certain IoU score.uncertainty/iou_<IoU score>/mce- Maximum Calibration Error (MCE) for a certain IoU score.uncertainty/iou_<IoU score>/nll- Negative Log Likelihood (NLL) for a certain IoU score.uncertainty/iou_<IoU score>/brier- Brier Score (BS) for a certain IoU score.fairness/rating_score- Rating score of detection fairness.fairness/iou_<IoU score>/demographic_parity_difference_score- Demographic parity difference for a certain IoU score.data_evaluation/rating_score- Rating score of dataset quality.
Note: the computation of many metrics depends on the selected IoU scores. These metrics are given for any specified IoU score.
- Diagrams:
The set of diagrams passed to MLflow depends on the selection of evaluation tasks. The following Matplotlib diagrams are passed to MLflow when running in classification mode:
performance/confusion_matrix_fig.svg- Classification confusion matrix showing the fraction of correct and false predictions w.r.t. available classes.uncertainty/uncertainty_reliability_diagram_fig.svg- Reliability diagram in the context of uncertainty quality/calibration evaluation showing the miscalibration w.r.t. a certain confidence level.fairness/selection_rate_diagram_<Sensitive Feature>_fig.svg- Selection rate diagram showing the disparity in the selection rate (aka recall) w.r.t. a certain sensitive feature.fairness/disparity_diagram_<Sensitive Feature>_fig.svg- Performance disparity diagram showing the disparity in over-/underestimation w.r.t. a certain sensitive feature.data_evaluation/cls_ratio_dataset_fig.svg- Bar chart showing the label distribution of ground truth classes within the evaluation dataset.data_evaluation/feature_<Sensitive Feature>_ratio_dataset_fig.svg- Bar chart showing the label distribution of sensitive feature information within the evaluation dataset w.r.t. a certain sensitive feature.
Note: the creation of some diagrams depends on the specified sensitive features (fairness evaluation). For multiple sensitive features, multiple diagrams are created.
The following Matplotlib diagrams are passed to MLflow when running in classification mode:
performance/bar_chart_detection_performance_iou_<IoU score>_fig.svg- Bar chart showing the detection performance metrics (precision, recall, AP) for each label within the dataset and for a certain IoU score.performance/precision_recall_curve_iou_<IoU score>_fig.svg- Precision-Recall curve for each label within the dataset and for a certain IoU score.uncertainty/uncertainty_reliability_diagram_iou_<IoU score>_fig.svg- Reliability diagram in the context of uncertainty quality/calibration evaluation showing the miscalibration w.r.t. a certain confidence level for a certain IoU score.uncertainty/uncertainty_reliability_diagram_xy_iou_<IoU score>_fig.svg- Reliability diagram n the context of object detection uncertainty quality/calibration evaluation showing the miscalibration w.r.t. the cx/cy position of detected objects (as a heatmap) for a certain IoU score.uncertainty/uncertainty_reliability_diagram_wh_iou_<IoU score>_fig.svg- Reliability diagram n the context of object detection uncertainty quality/calibration evaluation showing the miscalibration w.r.t. the width/height position of detected objects (as a heatmap) for a certain IoU score.fairness/selection_rate_diagram_<Sensitive Feature>_iou_<IoU score>_fig.svg- Selection rate diagram showing the disparity in the selection rate (aka recall) w.r.t. a certain sensitive feature for a certain IoU score.data_evaluation/target_data_distribution_fig.svg- Heatmap showing the distribution of real target objects w.r.t. their cx/cy position.data_evaluation/pred_data_distribution_fig.svg- Heatmap showing the distribution of predicted objects w.r.t. their cx/cy position.data_evaluation/feature_<Sensitive Feature>_ratio_dataset_fig.svg- Bar chart showing the label distribution of sensitive feature information within the evaluation dataset w.r.t. a certain sensitive feature.
Note: the computation of some diagrams depends on the selected IoU scores. These diagrams are drawn for any specified IoU score.
- Artifacts:
The following artifacts are passed to MLflow:
thetis-result.json- Thetis result JSON with all metrics and diagrams mentioned above.report.pdf- Thetis result PDF report with detailed rating scores, recommendations, metrics, and diagrams.
- Parameters:
thetis_args (ThetisArgs) – A data class containing all necessary arguments for the execution, including the API key, license file path, license XML string, and license key/signature.
- Returns:
Dictionary with the evaluation results, rating scores, and recommendations for the examined AI model.
- Raises:
ConnectionError – if API key is given but if connection to Thetis user management cannot be established. (raised by
thetiscore._licensing.license._License.__init__()).LicenseInvalidError – if API key is given but if API key is either invalid or a valid license cannot be found. (raised by
thetiscore._licensing.license._License.__init__()).ThetisInternalLicenseError – on any other unexpected error during API key validation. (raised by
thetiscore._licensing.license._License.__init__()).FileNotFoundError – if file with given license_file_path cannot be found (raised by
thetiscore._licensing.license._License.__init__()).RuntimeError – if license_file_path is given but file path is malformed (e.g., due to insufficient escaping of backslashes) (raised by
thetiscore._licensing.license._License.__init__()).RuntimeError – if parsing the given XML file failed (raised by
thetiscore._licensing.license._License.__init__()).RuntimeError – if parsing the given XML string failed (raised by
thetiscore._licensing.License.__init__()).RuntimeError – if neither ‘license_file_path’, ‘license_xml_str’, ‘license_key_and_signature’, THETIS_LICENSE, “license.dat” in working directory nor “license.dat” in user local home directory (OS specific) could be found (raised by
thetiscore._licensing.license._License.__init__()).AttributeError – if the key ‘Key’ cannot be found at the expected location (raised by
thetiscore._licensing.license._License.__init__()).AttributeError – if the key ‘Signature’ cannot be found at the expected location (raised by
thetiscore._licensing.license._License.__init__()).thetiscore.errors.LicenseInvalidError – if the passed license key/signature pair is invalid (raised by
thetiscore._licensing.license._License.verify()).RuntimeError – if the license Key XML cannot be parsed (raised by
thetiscore._licensing.license._License.verify()).AttributeError – if the key ‘ExpiryDate’ cannot be found at the expected location (raised by
thetiscore._licensing.license._License.verify()).thetiscore.errors.LicenseExpiredError – if the license has expired (raised by
thetiscore._licensing.license._License.verify()).TypeError – if config is neither str nor python dict (raised by
thetiscore._data.get_config()).SyntaxError – if the syntax of the user configuration is malformed according to the required configuration schema (raised by
thetiscore._data.get_config()).AttributeError – if classification or detection mode but key “distinct_classes” is missing or empty (raised by
thetiscore._data.get_config()).AttributeError – if classification mode but length of “distinct_classes” is lower than 2 (raised by
thetiscore._data.get_config()).AttributeError – if binary classification mode (length of “distinct_classes” is 2) but error_msg field “binary_positive_label” is missing (raised by
thetiscore._data.get_config()).AttributeError – if detection mode but key “task_settings/detection_bbox_format” is missing in user config. (raised by
thetiscore._data.get_config()).AttributeError – if “fairness” evaluation is active but no sensitive attribute has been specified (raised by
thetiscore._data.get_config()).AttributeError – if more than 2 distinct_classes are specified along with a “binary_positive_label” (raised by
thetiscore._data.get_config()).AttributeError – if “binary_positive_label” key is specified for any task other than classification (raised by
thetiscore._data.get_config()).AttributeError – if “distinct_classes” key is specified for a regression task (raised by
thetiscore._data.get_config()).RuntimeError – if the license Key XML cannot be parsed (raised by
thetiscore._licensing.license._License.verify_feature()).AttributeError – if the key ‘Features’ cannot be found at the expected location (raised by
thetiscore._licensing.license._License.verify_feature()).NotImplementedError – if “task” is not one of “classification”, “regression”, “detection” (raised by
thetiscore._data.get_input()).RuntimeError – if the given language identifier is unknown or not supported (raised by
thetiscore._language.Language._Language.__init__()).TypeError – if ‘predictions’ is not type pd.DataFrame (classification, raised by
thetiscore._data.get_input()).TypeError – if ‘annotations’ is not type pd.DataFrame (classification, raised by
thetiscore._data.get_input()).TypeError – if ‘predictions’ is not type dict (detection, raised by
thetiscore._data.get_input()).TypeError – if any value in ‘predictions’ dictionary is not type pd.DataFrame (detection, raised by
thetiscore._data.get_input()).TypeError – if ‘annotations’ is not type dict (detection, raised by
thetiscore._data.get_input()).TypeError – if any value in ‘annotations’ dictionary is not type pd.DataFrame (detection, raised by
thetiscore._data.get_input()).TypeError – if ‘description’ is not type dict
TypeError – if ‘description’ values are not type str
KeyError – if ‘description’ has unexpected or missing keys
RuntimeError – if the indices of the predicted and ground-truth data set do not match to each other. (classification, raised by
thetiscore._data.get_input()).ValueError – if “distinct_classes” in application config contains duplicates (classification or detection, raised by
thetiscore._data.get_input()).AttributeError – if the requested column “labels” does not exist within the predicted data set (classification or detection, raised by
thetiscore._data.get_input()).AttributeError – if the requested column “predictions” does not exist within the predicted data set (regression, raised by
thetiscore._data.get_input()).AttributeError – if the requested column “target” does not exist within the ground-truth data set (raised by
thetiscore._data.get_input()).ValueError – if the data type conversion to one of “int” or “str” of column “labels” failed within the predicted data set (classification or detection, raised by
thetiscore._data.get_input()).ValueError – if the data type conversion to “float” of column “predictions” failed within the predicted data set (regression, raised by
thetiscore._data.get_input()).ValueError – if the data type conversion to one of “int” or “str” of column “target” failed within the ground-truth data set (classification or detection, raised by
thetiscore._data.get_input()).ValueError – if the data type conversion to “float” of column “target” failed within the ground-truth data set (regression, raised by
thetiscore._data.get_input()).RuntimeError – if the arrays for predicted labels and ground-truth data set have different dtypes (classification or detection, raised by
thetiscore._data.get_input()).ValueError – if classes are found that have not been specified by ‘distinct_classes’ in the config file (classification or detection, raised by
thetiscore._data.get_input()).ValueError – if binary classification and “binary_positive_label” can not be found in “distinct_classes” (classification or detection, raised by
thetiscore._data.get_input()).ValueError – if binary classification and a complementary label to “binary_positive_label” can not be found in “distinct_classes” (classification or detection, raised by
thetiscore._data.get_input()).RuntimeError – if multi-class extraction failed and if more than 2 distinct classes are detected (binary classification, raised by
thetiscore._data.get_input()).AttributeError – if multi-class extraction failed and if the requested column “confidence” does not exist within the predicted data set (binary classification or detection, raised by
thetiscore._data.get_input()).ValueError – if multi-class extraction failed and if the data type conversion to “float” of column “confidence” failed within the predicted data set (binary classification or detection, raised by
thetiscore._data.get_input()).RuntimeError – if multi-class extraction failed and if the specified positive label within the application config cannot be found in the data set (binary classification, raised by
thetiscore._data.get_input()).AttributeError – if a sensitive feature is defined for a label that has not been found in the data set. (raised by
thetiscore._data.get_input()).RuntimeError – if uncertainty evaluation is active but no uncertainty information is given (regression, raised by
thetiscore._data.get_input()).ValueError – if column “variance” exists in predictions dataframe and if the data type conversion to “float” of column “variance” failed within the predicted data set (regression, raised by
thetiscore._data.get_input()).ValueError – if column “stddev” exists in predictions dataframe and if the data type conversion to “float” of column “stddev” failed within the predicted data set (regression, raised by
thetiscore._data.get_input()).AttributeError – if the requested column for a sensitive feature does not exist within the ground-truth data set (raised by
thetiscore._data.get_input()).ValueError – if the data type conversion to one of “int” or “str” failed for a requested sensitive feature within the ground-truth data set (raised by
thetiscore._data.get_input()).ValueError – if a sensitive feature has a missing or invalid entry (raised by
thetiscore._data.get_input()).ValueError – if a sensitive feature has less than 2 distinct labels (raised by
thetiscore._data.get_input()).RuntimeError – if bbox_format is not one of ‘xyxy’, ‘xywh’, or ‘cxcywh’ (detection, raised by:
thetiscore._data.get_input()).AttributeError – if the field ‘__meta__’ within the ground-truth data set annotations is completely missing (detection, raised by:
thetiscore._data.get_input()).AttributeError – if the meta information provided by ‘__meta__’ is missing for a certain image (detection, raised by:
thetiscore._data.get_input()).ValueError – if the image width or height information provided by ‘__meta__’ for a certain image are <= 0 (detection, raised by:
thetiscore._data.get_input()).ValueError – if bbox_format is ‘xyxy’ and xmin > xmax (detection, raised by:
thetiscore._data.get_input()).ValueError – if bbox_format is ‘xyxy’ and ymin > ymax (detection, raised by:
thetiscore._data.get_input()).ValueError – if bbox_format is ‘xywh’ or ‘cxcywh’ and width is negative (detection, raised by:
thetiscore._data.get_input()).ValueError – if bbox_format is ‘xywh’ or ‘cxcywh’ and height is negative (detection, raised by:
thetiscore._data.get_input()).RuntimeError – if the fields “weight” or “invert” are missing within a YAML file for a certain rating aspect (raised by
thetiscore._utils.get_rating_score()).RuntimeError – if “evaluation_spline” has been defined within a YAML file for a certain rating aspect but fieds “x” or “y” are missing. (raised by
thetiscore._utils.get_rating_score()).RuntimeError – if the “weight” parameters do not sum up to 1 during metric calculation. (raised by
thetiscore._utils.get_rating_score()).RuntimeError – if ‘rating_scores’ and ‘rating_aspects’ do not have the same length (raised by
thetiscore._utils.get_rating_recommendations()).ValueError – if the confidence for any prediction is greater than 1 (classification, raised by
thetiscore._data.get_input()).ValueError – if the confidence for any prediction is lower than 0 (classification, raised by
thetiscore._data.get_input()).ValueError – if there are any NaN or missing values in the required columns in either predictions or annotations (raised by
thetiscore._data.get_input()).NotImplementedError – if the bounding box matching strategy is not one of “exclusive”, “max” (raised by
thetiscore._data.get_input()).ValueError – if all prediction and annotation dataframes are empty, and task is set to object detection (raised by
thetiscore._data.get_input()).NotImplementedError – if detection_bbox_clipping is specified and is not one of ‘relative .%’ or ‘absolute .px’ (raised by
thetiscore._data.get_input()).RuntimeError – if detection_bbox_clipping value is not specified correctly (in case of relative or absolute px) (raised by
thetiscore._data.get_input()).RuntimeError – if task is detection and any boxes are out of bounds set by the detection_bbox_clipping parameter (raised by:
thetiscore._data.utils.get_input()).
Tensorboard format¶
Coming soon