Preparing input data ==================== The Thetis evaluation library only requires the output of an AI model on a dedicated evaluation dataset. Thus, the application requires the ground truth target labels (not the data itself!) and the according AI predictions. We give a detailed overview about the required data format in the following. Binary classification --------------------- In the case of binary classification, Thetis expects two instances of a `Pandas DataFrame `__ :code:`pd.DataFrame` representing the annotations and the model predictions on the dataset, respectively. Imagine that we have the variables :code:`annotations` and :code:`predictions` with annotations and predictions, respectively. If we print the content of :code:`annotations` to the Python console, it might output the following: .. code-block:: python >>> annotations target gender age img_00 person female adult img_01 person male child img_02 person female adult img_03 person female adult img_04 person female adult img_05 person female adult img_06 no person male adult img_07 no person female adult img_08 person female child img_09 person female child In this example, we have the available target classes "person" and "not person" with the sensitive attributes "gender" and "age". The target labels must always be given in the column :code:`target`. Furthermore, each sensitive attribute must be given with its respective name as column name. Accordingly, the output of :code:`predictions` might look like the following: .. code-block:: python >>> predictions labels confidence img_00 person 0.992300 img_01 person 0.962620 img_02 no person 0.146000 img_03 person 0.795490 img_04 person 0.897310 img_05 no person 0.247210 img_06 no person 0.001412 img_07 no person 0.000150 img_08 person 0.970410 img_09 person 0.931941 The required column :code:`labels` holds the predicted label for each item in the dataset, whereas the required column :code:`confidence` represents the (binary) label confidence/uncertainty estimated by the AI model. **Important:** the :code:`confidence` refers to the :code:`binary_positive_label` specified in the `application config <#>`__. The uncertainty for the negative class ("no person" in this case) is given by :code:`1 - confidence`. **Note:** the indices of the DataFrames :code:`annotations` and :code:`predictions` must match to each other. Multi-class classification -------------------------- If you are working in a multi-class classification setting (setting "task" to "classification" with more than 2 entries in "distinct_classes" in the `application config <#>`__), Thetis also expects two Pandas DataFrames :code:`annotations` and :code:`predictions` representing the ground truth annotations and the according AI predictions, respectively. Similar to the binary classification case, the DataFrame :code:`annotations` must consist of the columns :code:`target` as well as an own column for each specified sensitive attribute: .. code-block:: python >>> annotations target gender age img_00 person female adult img_01 person male child img_02 car None None img_03 bicycle None None img_04 car None None img_05 person female child img_06 car None None img_07 person female adult img_08 person male adult img_09 bicycle None None **Note:** in this example, the sensitive attributes "gender" and "age" are only valid for the target class "person". This must be specified in the `fairness section of the application config <#>`__. For all other entries, the missing entries are marked by passing "None". The respective AI predictions given by :code:`predictions` are given in a similar way compared to binary classification. A column :code:`labels` is required, representing the predicted label of each individual sample. However, an uncertainty/confidence is required for each possible class (e.g., the output of a Softmax activation function in the context of neural networks). The columns for the confidence must follow the naming convention :code:`confidence_