PRIDE2 Server test results

The PRIDE2 method and the corresponding server were tested using a previously published and an in-house test set. The test sets are as follows (click on the links to see the obtained results):

1. Test set based on that published by Novotny et al. (Novotny,M., Madsen,D. and Kleywegt, G.J. (2004): Evaluation of Protein Fold Comparison Servers. Proteins, 54, 260-270).
2. A set of newly determined structures retrieved from PDB on 29 Dec 2004 using the selection criteria below:
   
   • Deposited after Sep 1, 2004
   • Released after Dec 1, 2004
   • 90% homology filtered
   • For X-ray structures resolution higher than < 2 Angstrom (55 structures)
   • NMR structures (18 structures)
3. In addition, a quick comparison of the PRIDE and PRIDE2 methods is presented

The 'real' test was feeding the full PDB structures (listed in Table 2. of the paper) to the server using default settings and, where necessary, settings optimized for best performance with the given query. The result was considered positive if the server reported a structure falling into the same CATH 'H' level as the query (using the classification of Table 2 in the paper, it should be noted that a number of the structures contains additonal domains listed in CATH) excluding self hits. The table below summarizes the results for each query and details the optimizations that were necessary to obtain positive hits.
We note here a few differences in our data set compared to the original one used by Novotny et al.: the class 3.10.70 in the paper has changed in CATH and is denoted 3.10.50 in version 2.5.1; 1hyl was considered misspelled as 1hy1 is actually in class 1.10.40; 1c3u is not in class 1.10.164 but in 10.10.40.
The 'default' server settings are: H-level filtered CATHselect search, Window/Slide: 160/80 and reporting the 10 best structures for each query (segment). The rank reported corresponds to hits for the segment the correct hit was found for. The last unit of the classification (L#) denotes the lenght of the domain. Optimizations are noted as differences compared to the default settings (and listed also for some positive but weak hits with the default settings).

PDB ID	CATH H level	Rank of 1st correct hit	PRIDE2 of 1st correct hit	CATH name/classification of correct hit	Optimizations/results obtained
All-α structures
1RLR	1_10_40_20	-	-		Window/Slide: 80/40, full database search: 1st hit: 0.907 3r1rC1 1_10_40_20_1_1_1_L86
1C3U	1_10_40_30	-	-		Window/Slide: 80/40, 5th hit: 0.544 1i0aD3 1_10_40_30_2_3_1_L72
1AUW	1_10_40_30	3	0.430	1hy0B3 1_10_40_30_2_2_1_L72	Window/Slide: 70/30, 1st hit: 0.991 1k7wD3 1_10_40_30_2_1_1_L72
1FUR	1_10_40_30	1	0.999	2fusB3 1_10_40_30_1_1_2_L51
1HY1	1_10_40_30	1	0.429	1hy0B3 1_10_40_30_2_2_1_L72	Window/Slide: 72/36 , 1st hit: 0.728 1dcnD3 1_10_40_30_2_1_2_L72
1I0A	1_10_40_30	2	0.420	1k62B3 1_10_40_30_2_4_1_L72	Window/Slide: 72/36, full database search: 1st hit: 0.943 1k62B3 1_10_40_30_2_4_1_L72
1JSW	1_10_40_30	5	0.785	1yfm03 1_10_40_30_1_2_1_L52
1YFM	1_10_40_30	1	0.899	2fusA3 1_10_40_30_1_1_1_L52
1AQ6	1_10_164_10	-	-		Window/Slide: 76/38, full database search: 1st hit: 0.804 1qq7B2 1_10_164_10_1_2_3_L75
1ZRN	1_10_164_10	-	-		Window/Slide: 80/40, full database search, 2nd hit: 0.809 1jud02 1_10_164_10_1_1_1_L75
1JUD	1_10_164_10	-	-		Window/Slide: 76/38, full database search: 8th hit: 0.730 1zrn02 1_10_164_10_1_3_1_L75
1FEZ	1_10_164_20	-	-		Unique at this 'H' level (no non-self hits can be expected)
1B3U	1_25_10_10	4	0.503	1m5nS0 1_25_10_10_6_1_5_L485
1BK6	1_25_10_10	2	0.980	1ee5A0 1_25_10_10_3_1_4_L421
1GCJ	1_25_10_10	2	0.961	1ibrB0 1_25_10_10_6_1_1_L458
1IAL	1_25_10_10	2	0.994	1pjmB0_1_25_10_10_3_2_3_L427
1IBR	1_25_10_10	3	0.962	1o6pB0_1_25_10_10_6_1_3_L441
1QBK	1_25_10_10				Finds only itself
2BCT	1_25_10_10	2	0.950	1jpwC0 1_25_10_10_2_1_9_L502
All-β structures
1CI0	2_30_110_10	3	0.902	1g79A0 2_30_110_10_1_2_2_L199
1DNL	2_30_110_10	1	1.000	1g79A0 2_30_110_10_1_2_2_L199
1EJE	2_30_110_20	2	0.737	1i0sA0_2_30_110_20_2_1_1_L161
1I0R	2_30_110_20	1	1.000	1i0sA0 2_30_110_20_2_1_1_L161
1A33	2_40_100_10	1	1.000	1c5fE0 2_40_100_10_2_4_1_L174
1AWQ	2_40_100_10	1	1.000	1m9eB0 2_40_100_10_2_1_6_L162
1CYN	2_40_100_10	2	1.000	2rmcG0 2_40_100_10_2_3_1_L182
1DYW	2_40_100_10	1	1.000	1e8kA0 2_40_100_10_2_6_1_L172
1IHG	2_40_100_10	1	0.998	1iipA1 2_40_100_10_2_8_1_L184
1LOP	2_40_100_10	2	1.000	2nul00 2_40_100_10_1_2_2_L163
1QNG	2_40_100_10	2	1.000	1qnhA0 2_40_100_10_2_7_2_L169
1QOI	2_40_100_10	2	0.984	1c5fI0 2_40_100_10_2_4_4_L173
2RMC	2_40_100_10	2	1.000	1cynA0 2_40_100_10_2_2_1_L178
1CIY	2_100_10_10	2	0.635	1dlc03 2_100_10_10_1_1_1_L197
1DLC	2_100_10_10	2	0.912	1ji6A3 2_100_10_10_3_1_1_L199
1VMO	2_100_10_20	-	-		Unique at this 'H' level (no non-self hits can be expected)
1C3K	2_100_10_30	2	1.000	1c3nA0 2_100_10_30_2_1_1_L144
1JAC	2_100_10_30	1	1.000	1m26G0 2_100_10_30_1_1_1_L133
1JOT	2_100_10_30	2	0.999	1m26G0 2_100_10_30_1_1_1_L133
Mixed α-β structures
1GRJ	3_10_50_30	-	-		Unique at this 'H' level (no non-self hits can be expected)
1BKF	3_10_50_40	1	1.000	2fke00 3_10_50_40_1_1_1_L107
1PBK	3_10_50_40	1	0.811	1n1aB0 3_10_50_40_1_8_1_L121
1ROT	3_10_50_40	1	0.971	1rou00_3_10_50_40_1_4_1_L118
1YAT	3_10_50_40	2	0.993	1bl4B0 3_10_50_40_1_1_5_L107
1CFR	3_40_91_10	2	0.795	1knvB0 3_40_91_10_3_1_2_L291
1BHM	3_40_91_20	2	1.000	2bamB0 3_40_91_20_1_1_4_L210
1D2I	3_40_91_20	1	1.000	1dfmB0 3_40_91_20_2_1_2_L213
1FOK	3_40_91_30	2	0.986	2fokA3 3_40_91_30_1_1_2_L175
1AXC	3_70_10_10	4	0.956	1plr00 3_70_10_10_1_1_1_L258
1GE8	3_70_10_10	2	0.995	1iz4A0_3_70_10_10_2_1_3_L241
1PLQ	3_70_10_10	1	0.978	1plr00 3_70_10_10_1_1_1_L258
1B77	3_70_10_20	1	1.000	1czdC0 3_70_10_20_1_2_1_L228
1CZD	3_70_10_20	2	0.994	1b8hC0 3_70_10_20_1_1_1_L228
1DML	3_70_10_30	-	-		Unique at this 'H' level (no non-self hits can be expected)
Folds with few secondary structures
1B2I	2_40_20_10	2	0.894	2pk400 2_40_20_10_1_1_1_L80
1CEA	2_40_20_10	1	0.994	2pf202 2_40_20_10_1_10_1_L82
1KDU	2_40_20_10	2	0.754	1tpkC0 2_40_20_10_1_3_1_L88
1KIV	2_40_20_10	2	0.999	4kiv00 2_40_20_10_1_7_1_L79
1KRN	2_40_20_10	2	0.994	1i71A0 2_40_20_10_1_11_1_L83
1PK4	2_40_20_10	1	0.999	3kiv00 2_40_20_10_1_7_3_L79
1PML	2_40_20_10	1	0.919	1tpkC0_2_40_20_10_1_3_1_L88
5HPG	2_40_20_10	2	0.961	2pk400 2_40_20_10_1_1_1_L80

It is apparent that the group 1_10_164 performs remarkably poor. This is partly because the size of the domains is less than half of the default window size (although this seems not a serious hindrance in other groups!) and the structures in CATHselect at these 'H' levels are not similar enough to yield a group score high enough for effective H-level filtered search.
It should be noted that if self-hits are considered, all queries were correctly identified (if the query is in CATHselect, it is the first hit, that is why in many cases the 2nd, 3rd.. hit is listed). In another test, using all domain structures (instead of the full PDB files) listed in CATH 2.5.1 and belonging to the groups considered by Novotny et al, a success rate of > 95% was obtained by the H-level filtered method and all remaining structures were found by full CATHselect search (counting self-hits as positive).

The sequences of the new structures were submitted to a BLAST search against the domain sequences in CATH2.5.1. The significant BLAST hits (the best one for each query and domain hit) were then compared to the best PRIDE2 hit. From the 28 different (significant) BLAST hits 19 were reproduced (i.e. a best hit at the same 'H' level was found) using Window/Slide 160/80 and 25 by using 100/50.

Group used: 2_60_40_10 (1115 domains)	Group used: 3_40_50_720 (308 domains)
Full ROC curve	Full ROC curve
Upper part of ROC curve	Upper part of ROC curve