Calculate evaluation metrics for bulk RNA-Seq samples from test data

Source: R/evalMetrics.R

Calculate evaluation metrics for bulk RNA-seq samples from test data to understand model performance. By default, absolute error (AbsErr), proportional absolute error (ppAbsErr), squared error (SqrErr) and proportional squared error (ppSqrErr) are calculated for each test sample. In addition, each of these metrics is aggregated using their mean values according to three criteria: each cell type (CellType), probability bins in ranges of 0.1 (pBin) and number of different cell types present in the sample nCellTypes. Finally, the process is repeated only considering bulk samples (filtering out single-cell profiles from the evaluation). The evaluation metrics will be available in the test.deconv.metrics slot of the DigitalDLSorterDNN object (trained.model slot of the DigitalDLSorter object).

calculateEvalMetrics(object, metrics = c("MAE", "MSE"))

Arguments

object

DigitalDLSorter object with a trained model in the trained.model slot and the actual cell proportions of pseudo-bulk samples in prob.cell.matrix slot.

metrics

Metrics used to evaluate the model performance. Mean absolute error ("MAE") and mean squared error ("MSE") by default.

Value

A DigitalDLSorter object with the trained.model slot containing a DigitalDLSorterDNN object with the test.deconv.metrics slot. The last contains the metrics calculated.

Examples

if (FALSE) { # \dontrun{
set.seed(123)
sce <- SingleCellExperiment::SingleCellExperiment(
  assays = list(
    counts = matrix(
      rpois(30, lambda = 5), nrow = 15, ncol = 20,
      dimnames = list(paste0("Gene", seq(15)), paste0("RHC", seq(20)))
    )
  ),
  colData = data.frame(
    Cell_ID = paste0("RHC", seq(20)),
    Cell_Type = sample(x = paste0("CellType", seq(6)), size = 20,
                       replace = TRUE)
  ),
  rowData = data.frame(
    Gene_ID = paste0("Gene", seq(15))
  )
)
DDLS <- createDDLSobject(
  sc.data = sce,
  sc.cell.ID.column = "Cell_ID",
  sc.gene.ID.column = "Gene_ID",
  sc.filt.genes.cluster = FALSE, 
  sc.log.FC = FALSE
)
probMatrixValid <- data.frame(
  Cell_Type = paste0("CellType", seq(6)),
  from = c(1, 1, 1, 15, 15, 30),
  to = c(15, 15, 30, 50, 50, 70)
)
DDLS <- generateBulkCellMatrix(
  object = DDLS,
  cell.ID.column = "Cell_ID",
  cell.type.column = "Cell_Type",
  prob.design = probMatrixValid,
  num.bulk.samples = 50,
  verbose = TRUE
)
# training of DDLS model
tensorflow::tf$compat$v1$disable_eager_execution()
DDLS <- trainDDLSModel(
  object = DDLS,
  on.the.fly = TRUE,
  batch.size = 15,
  num.epochs = 5
)
# evaluation using test data
DDLS <- calculateEvalMetrics(
  object = DDLS
)
} # }