Grand Challenge 2 Evaluation Results - Obsolete

Spreadsheets (.csv)

Graphs

User Submitted Files

Evaluation Methods
Pose predictions were evaluated in terms of symmetry-corrected root-mean-square deviations (RMSD, Å). In the graphs, the pull-down menu allows display of the results for all submissions for each compound (e.g., FXR_1), or results for all submissions averaged over all compounds (AVG). The three buttons following the menu in the graphs allow display of the RMSD values for lowest RMSD pose among the maximum of five in each submission (Best); of the average RMSD value across all five (Average); or of the RMSD value for the top-scoring pose in each submission (Pose 1). The csv files provide the statistics averaged over ligands. Note that FXR_33 is excluded from pose prediction evaluations because of crystal artifacts found in the co-crystal structure.

The rankings of compounds by affinity and the free energy calculations were evaluated in terms of the Kendall's tau and Spearman rho statistics, and the free energy calculations were additionally evaluated in terms of centered RMSDs (kcal/mol) and Pearson's r. Uncertainties in these statistics (e.g., Kendall Tau Err in csv files) were obtained by recomputing them in 10,000 rounds of resampling with replacement, where, in each sample, the experimental IC50 data were randomly modified based on the experimental uncertainties. The graphical representations provided here do not include Spearman's rho or Pearson's r.

Further details of these procedures and results will be provided in an overview paper in the special issue of JCAMD.

All scripts used to evaluate submissions are publicly available on Github.

Note regarding racemic compounds
The instructions for this challenge stated that some of the compounds were tested by Roche as 50:50 racemates, and that there was evidence that the activity derived from only one of the enantiomers. For a 50:50 racemate, this would mean the raw IC50 is twice the IC50 of the active enantiomer. However, the instructions stated that participants should provide predictions that could be compared directly with the raw data. For these compounds, therefore, we have used your predictions without any adjustment to the raw experimental IC50 values. The compounds treated in this manner are as follows: FXR_37, FXR_39, FXR_40, FXR_42, FXR_50, FXR_51, FXR_52, FXR_53, FXR_54, FXR_55, FXR_56, FXR_57, FXR_58, FXR_59, FXR_60, FXR_61, FXR_62, FXR_63, FXR_64, FXR_66, FXR_67, FXR_68, FXR_69, FXR_70, FXR_71, FXR_72.

By our accident, the instructions did not state that some other compounds also were tested experimentally as racemates. The following compounds were tested as 50:50 mixtures: FXR_6, FXR_7, FXR_8, FXR_9, FXR_13, FXR_18, FXR_20, FXR_22, FXR_25, FXR_31, FXR_35, FXR_36. For these compounds, we have divided the experimental IC50 values by 2 before comparing with your predictions. In addition, FXR_22 was tested as a 25:25:25:25 mixture of stereoisomers, so we have divided its experimental IC50 value by 4 before comparing with your predictions.

Revision 2017-04-20 : Error statistics were updated after we received further details regarding uncertainties in the experimental IC50 values, as well as a small correction. The originally posted results are below.

Obsolete Results

Spreadsheets (.csv)

Graphs