Grand Challenge 2 Evaluation Results

Overviews

User Submitted Files

Pose predictions were evaluated in terms of symmetry-corrected root-mean-square deviations (RMSD, Å). In the graphs, the pull-down menu allows display of the results for all submissions for each compound (e.g., FXR_1), or results for all submissions averaged over all compounds (AVG). The three buttons following the menu in the graphs allow display of the RMSD values for lowest RMSD pose among the maximum of five in each submission (Best); of the average RMSD value across all five (Average); or of the RMSD value for the top-scoring pose in each submission (Pose 1). The csv files provide the statistics averaged over ligands. Note that FXR_33 is excluded from pose prediction evaluations because of crystal artifacts found in the co-crystal structure.

The rankings of compounds by affinity and the free energy calculations were evaluated in terms of the Kendall's tau and Spearman rho statistics, and the free energy calculations were additionally evaluated in terms of centered RMSEs (kcal/mol) and Pearson's r. Uncertainties in these statistics (e.g., Kendall Tau Err in csv files) were obtained by recomputing them in 10,000 rounds of resampling with replacement, where, in each sample, the experimental IC50 data were randomly modified based on the experimental uncertainties. The graphical representations provided here do not include Spearman's rho or Pearson's r.

All free energy submissions were alchemical approaches that used explicit solvent free energy methods and provided relative binding free energies between pairs of ligands across the two challenge Stages and the two free energy sets, except for one submission, xk67c, which computed the absolute binding free energy of each separate ligand. In addition to the free energy methods, other approaches were applied to the free energy sets as part of the free energy component of the challenge; these included methods based on scoring functions, force fields combined with implicit solvent, and electronic structure calculations with implicit solvent.

The two free energy ligand sets are part of the full series of 102 ligands, for which a variety of additional methods were tested in the affinity ranking component of the challenge. This meant that we could extract the predictions for the free energy sets of compounds from the larger set of ranking and scoring methods used for the 102 compound set, and thus put the more detailed free energy methods into the context of the faster methods that were applied to the full set of 102 ligands. Therefore, in the accompanied graphs and tables, RMSEc statistics are calculated for predictions from scoring methods with free energy estimates in the affinity ranking component of the challenge, and free energy methods in the free energy component of the challenge; and Kendall?s tau statistics are calculated for predictions from all scoring and free energy methods.

Further details of these procedures and results will be provided in an overview paper in the special issue of JCAMD.

All scripts used to evaluate submissions are publicly available on Github.

The instructions for this challenge stated that some of the compounds were tested by Roche as 50:50 racemates, and that there was evidence that the activity derived from only one of the enantiomers. For a 50:50 racemate, this would mean the raw IC50 is twice the IC50 of the active enantiomer. However, the instructions stated that participants should provide predictions that could be compared directly with the raw data. For these compounds, therefore, we have used your predictions without any adjustment to the raw experimental IC50 values. The compounds treated in this manner are as follows: FXR_37, FXR_39, FXR_40, FXR_42, FXR_50, FXR_51, FXR_52, FXR_53, FXR_54, FXR_55, FXR_56, FXR_57, FXR_58, FXR_59, FXR_60, FXR_61, FXR_62, FXR_63, FXR_64, FXR_66, FXR_67, FXR_68, FXR_69, FXR_70, FXR_71, FXR_72.

By our accident, the instructions did not state that some other compounds also were tested experimentally as racemates. The following compounds were tested as 50:50 mixtures: FXR_6, FXR_7, FXR_8, FXR_9, FXR_13, FXR_18, FXR_20, FXR_22, FXR_25, FXR_31, FXR_35, FXR_36. For these compounds, we have divided the experimental IC50 values by 2 before comparing with your predictions. In addition, FXR_22 was tested as a 25:25:25:25 mixture of stereoisomers, so we have divided its experimental IC50 value by 4 before comparing with your predictions.

Revision 2017-07-7 : Free energy evaluations were updated with ranking and scoring methods from the affinity ranking component of the challenge as detailed above, in order to put the more detailed free energy methods into the context of the faster methods that were applied to the full set of 102 ligands. The originally posted results are below.

Originally posted results

Revision 2017-04-20 : Error statistics were updated after we received further details regarding uncertainties in the experimental IC50 values, as well as a small correction. The originally posted results are below.

Obsolete Results