Evaluation

ninolearn.learn.evaluation.evaluation_correlation(model_name, variable_name='mean')[source]

Evaluate the model using the correlation skill for the full time series.

Parameters

  • model_name (str) – The name of the model.

  • variable_name (str) – The name of the variable which shell be evaluated against the ONI prediction.

Returns

The correlation skill for the 0, 3, 6, 9, 12 and 15-month lead time and the corresponding p values.

ninolearn.learn.evaluation.evaluation_decadal_correlation(model_name, variable_name='mean')[source]

Evaluate the model in the decades 1963-1971, 1972-1981, …, 2012-2017 using the correlation skill-

Parameters

  • model_name (str) – The name of the model.

  • variable_name (str) – The name of the variable which shell be evaluated against the ONI prediction.

Returns

The correlation skill for the 0, 3, 6, 9, 12 and 15-month lead time and the corresponding p values for the respective decades. The returned arrays have the shape (lead time, decades).

ninolearn.learn.evaluation.evaluation_decadal_nll(model_name, mean_name='mean', std_name='std')[source]

Evaluate the model in the decades 1963-1971, 1972-1981, …, 2012-2017 using the negative log-likelihood.

ninolearn.learn.evaluation.evaluation_decadal_srmse(model_name, variable_name='mean', decadal=None)[source]

Evaluate the model in the decades 1963-1971, 1972-1981, …, 2012-2017 using the standardized RMSE.

Parameters

  • model_name (str) – The name of the model.

  • variable_name (str) – The name of the variable which shell be evaluated against the ONI prediction.

Returns

The SRMSE for the 0, 3, 6, 9, 12 and 15-month lead time respective decades. The returned array has the shape (lead time, decades).

ninolearn.learn.evaluation.evaluation_nll(model_name, mean_name='mean', std_name='std')[source]

Evaluate the model using the negativ log-likelihood skill for the full time series.

ninolearn.learn.evaluation.evaluation_seasonal_correlation(model_name, variable_name='mean', background='all')[source]

Evaluate the model in different seasons using the correlation skill.

Parameters

  • model_name (str) – The name of the model.

  • variable_name (str) – The name of the variable which shell be evaluated against the ONI prediction.

Returns

The correlation skill for different seasons and the 0, 3, 6, 9, 12 and 15-month lead time and the corresponding p values for the respective seasons and lead times. The returned arrays have the shape (lead time, season). The season corresponding to the the array entry [:,0] is DJF and to [:,1] is JFM (and so on).

ninolearn.learn.evaluation.evaluation_seasonal_srmse(model_name, variable_name='mean', background='all')[source]

Evaluate the model in different seasons using the standardized RMSE.

Parameters

  • model_name (str) – The name of the model.

  • variable_name (str) – The name of the variable which shell be evaluated against the ONI prediction.

Returns

The SRMSE for different seasons and the 0, 3, 6, 9, 12 and 15-month lead times. The returned arrays have the shape (lead time, season). The season corresponding to the the array entry [:,0] is DJF and to [:,1] is JFM (and so on).

ninolearn.learn.evaluation.evaluation_srmse(model_name, variable_name='mean')[source]

Evaluate the model using the standardized root-mean-squarred error (SRMSE) for the full time series. Standardized means that the the the RMSE of each season is divided by the corresponding standard deviation of the ONI in that season (standard deviation has a seasonal cycle). Then, these seasonal SRMSE averaged to get the SRMSE of the full time series..

Parameters

  • model_name (str) – The name of the model.

  • variable_name (str) – The name of the variable which shell be evaluated against the ONI prediction.

Returns

The standardized RMSE for the 0, 3, 6, 9, 12 and 15-month lead time.

Skill measures

ninolearn.learn.skillMeasures.inside_fraction(y, pred_mean, pred_std, std_level=1)[source]

Returns the fraction of how much of the true observation is in the confindence interval.

Parameters

  • ytrue – The true observation.

  • pred_mean (array_like) – The mean of the prediction.

  • pred_std (array_like) – The standard deviation of the prediction.

  • std_level (int) – The standard deviation of the confidence interval.

Return type

float

Returns

The fraction of the observation that is in the confidence interval.

ninolearn.learn.skillMeasures.mean_srmse(y, predict, time)[source]

Mean SRMSE.

Parameters

  • y (array_like) – The true observation.

  • pred (array_like) – The prediction.

  • y – The corresponding time array for the target season.

Return type

float

Returns

The mean SRMSE value.

ninolearn.learn.skillMeasures.nll_gaussian(y, pred_mean, pred_std)[source]

Negative - log -likelihood for the prediction of a gaussian probability

ninolearn.learn.skillMeasures.rmse(y, predict)[source]

The root-mean-squarred error (RMSE) for a given observation and prediction.

Parameters

  • y (array_like) – The true observation.

  • pred (array_like) – The prediction.

Return type

float

Returns

The RMSE value

ninolearn.learn.skillMeasures.seasonal_correlation(y, pred, time)[source]

Pearson correlation coefficient for each season. This function uses the scipy.stats.pearsonr function.

Parameters

  • y (array_like) – The true observation.

  • pred (array_like) – The prediction.

  • y – The corresponding time array for the target season.

Return type

array_like

Returns

r, p. Returns the Pearson correlation coefficent (r) and the correspondarray of p-value. Both have length 12 (values for each season).

ninolearn.learn.skillMeasures.seasonal_nll(y, pred_mean, pred_std, time)[source]

Negative log-likelihood (NLL) for each season.

Parameters

  • y (array_like) – The true observation.

  • pred (array_like) – The prediction.

  • y – The corresponding time array for the target season.

Return type

array_like

Returns

Returns the NLL for each season. Array has length 12 (value for each season).

ninolearn.learn.skillMeasures.seasonal_srmse(y, pred, time)[source]

Standardized RMSE (RMSE) for each season. Standardized means in this case that the RMSE is divided by the standard deviation of the correpsonding season.

Parameters

  • y (array_like) – The true observation.

  • pred (array_like) – The prediction.

  • y – The corresponding time array for the target season.

Return type

array_like

Returns

Returns the SRMSE for each season. Array has length 12 (value for each season).