R/iucnn_modeltest.R
iucnn_modeltest.RdTakes as input features produced with iucnn_prepare_features
and labels produced with iucnn_prepare_labels, as well as a
path to a log-file where results for each tested model will be stored. All
available options are identical to the iucnn_train_model
function and can be provided as vectors, e.g. dropout_rate =
c(0.0,0.1,0.3) and n_layers = c('30','40_20','50_30_10').
iucnn_modeltest will then iterate through all possible permutations of
the provided hyperparameter settings, train a separate model for each
hyperparameter combination, and store the results in the provided log-file.
iucnn_modeltest(
x,
lab,
logfile = "model_testing_logfile.txt",
model_outpath = "modeltest",
mode = "nn-class",
cv_fold = 1,
test_fraction = 0.2,
n_layers = c("50_30_10", "30"),
dropout_rate = c(0, 0.1, 0.3),
use_bias = TRUE,
balance_classes = FALSE,
seed = 1234,
label_stretch_factor = 1,
label_noise_factor = 0,
act_f = "relu",
act_f_out = "auto",
max_epochs = 5000,
patience = 200,
mc_dropout = TRUE,
mc_dropout_reps = 100,
randomize_instances = TRUE,
rescale_features = FALSE,
init_logfile = TRUE,
recycle_settings = FALSE
)a data.set, containing a column "species" with the species names, and subsequent columns with different features.
an object of the class iucnn_labels, as generated by
iucnn_prepare_labels containing the labels for all species.
character string. Define the filepath/name for the output log-file.
the path where to save the results on disk
character string. Choose between the IUCNN models "nn-class" (default, tensorflow neural network classifier), "nn-reg" (tensorflow neural network regression), or "bnn-class" (Bayesian neural network classifier)
integer (default=1). When setting cv_fold > 1,
iucnn_train_model will perform k-fold cross-validation. In this case, the
provided setting for test_fraction will be ignored, as the test
size of each CV-fold is determined by the specified number provided here.
numeric. The fraction of the input data used as test set.
character string. Define number node per layer by providing a character string where the number of nodes for each layer are separated by underscores. E.g. '50_30_10' (default) will train a model with 3 hidden layers with 50, 30, and 10 nodes respectively. Note that the number of nodes in the output layer is automatically determined based on the number of unique labels in the training set.
numeric. This will randomly turn off the specified fraction of nodes of the neural network during each epoch of training making the NN more stable and less reliant on individual nodes/weights, which can prevent over-fitting (only available for modes nn-class and nn-reg). See mc_dropout setting explained below if dropout shall also be applied to the predictions.
logical (default=TRUE). Specifies if a bias node is used in the first hidden layer.
logical (default=FALSE). If set to TRUE,
iucnn_train_model will perform supersampling of the training instances to
account for uneven class distribution in the training data. In case of
training an bnn-class model, choosing this option will add the estimation
of class weights instead, to account for class imbalances.
integer. Set a starting seed for reproducibility.
numeric (only for mode nn-reg). The provided value will be applied as a factor to stretch or compress the labels before training a regression model. A factor smaller < 1.0 will compress the range of labels, while a factor > 1 will stretch the range.
numeric (only for mode nn-reg). Add specified amount of random noise to the input labels to give the categorical labels a more continuous spread before training the regression model. E.g. a value of 0.2 will redraw a label of a species categorized as Vulnerable (class=2) randomly between 1.8 and 2.2, based on a uniform probability distribution.
character string. Specifies the activation
function should be used in the hidden layers.
Available options are: "relu", "tanh", "sigmoid", or "swish" (latter only for
bnn-class). If set to 'auto' (default), iucnn_train_model will pick a reasonable
default ('relu' for nn-class or nn-reg, and 'swish' for bnn-class).
character string. Similar to act_f, this specifies the activation function for the output layer. Available options are "softmax" (nn-class, bnn-class), "tanh" (nn-reg), "sigmoid" (nn-reg), or no activation function "" (nn-reg). When set to "auto" (default), a suitable output activation function will be chosen based on the chosen mode ('softmax' for nn-class or bnn-class, 'tanh' for nn-reg).
integer. The maximum number of epochs.
integer. Number of epochs with no improvement after which training will be stopped.
logical. If set to TRUE, the predictions (including the
validation accuracy) based on a model trained with a dropout fraction > 0
will reflect the stochasticity introduced by the dropout method (MC dropout
predictions). This is e.g. required when wanting to predict with a specified
accuracy threshold (see target_acc option in iucnn_predict_status).
This option is activated by default when chosing a dropout_rate > 0, unless
it is manually set to FALSE here.
integer. The number of MC iterations to run when predicting validation accuracy and calculating the accuracy-threshold table required for making predictions with an accuracy threshold. The default of 100 is usually sufficient, larger values will lead to longer computation times, particularly during model testing with cross-validation.
logical. When set to TRUE (default) the instances will be shuffled before training (recommended).
logical. Set to TRUE if all feature values shall be rescaled to values between 0 and 1 prior to training (default=FALSE).
logical (default=TRUE). If set to TRUE,
modeltest_iucnn will attempt to initiate a new log-file under the
provided path, possibly overwriting already existing model-testing results
stored in the same location. Set to FALSE if instead you want to append to
an already existing log-file.
logical (default=FALSE). If set to TRUE,
iucnn_modeltest will read the log-file stored at the path specified
under the "logfile" argument and run model-testing for the input features
and labels using the same models stored in that file. This setting can be
useful when e.g. wanting to test the same models for different sets of input
data.
outputs a data.frame object containing stats and settings of all tested models.
See vignette("Approximate_IUCN_Red_List_assessments_with_IUCNN")
for a tutorial on how to run IUCNN.
if (FALSE) {
data("training_occ") #geographic occurrences of species with IUCN assessment
data("training_labels")# the corresponding IUCN assessments
# 1. Feature and label preparation
features <- iucnn_prepare_features(training_occ, type = "geographic") # Training features
labels_train <- iucnn_prepare_labels(training_labels, features) # Training labels
# Model-testing
mod_test <- iucnn_modeltest(x = features,
lab = labels_train,
logfile = "model_testing_results-2.txt",
model_outpath = "iucnn_modeltest-2",
mode = "nn-class",
dropout_rate = c(0.0, 0.1, 0.3),
n_layers = c("30", "40_20", "50_30_10"),
cv_fold = 2,
init_logfile = TRUE)
}