|
Experiments in computer chess
The value of depth and diminishing return effects.
For this purpose we took ProDeo 1.74 playing depth based matches of (only) 800 games (4 x 200) on a Quad PC using ProDeo's parameter:
|
[PLY=x] |
x is the base depth for the middle game phase and gradually increased when the game
moves into different phases. |
When for example using [PLY = 8] then:
- All middle game positions will be played at depth 8.
- When Queens are off the base depth (8) is raised by 1 and becomes 9.
- Normal endgames will be played with depth 10.
- Simple endgames (rook endings) the base search depth increases with 3 and becomes 11.
- Light piece endings are searched with +4 thus 12 plies deep.
- And finally pawn endings at 13 plies.
- We make one exception for Queen endings, the depth increase is only 1, thus 9 plies because of the many check extensions that usually play a role in Queen endings.
This all to ensure that games based on depth will relative closely emulate the same depth behavior as playing games on time control. The system has the dvantage that it avoids any intereference from outside. Contrary to time control games depth based games always end the same way while time control based games are dependent on the capriciousness of the operating system (Windows and friends) and user interferences doing things into the background. As such we are excluding randomness in a process that is already so sensitive to randomness.
The error margin of 800 games is quite high but since we are dealing with big elo differences it's accetable to allow fluctuations of 1-2% max. For that purpose each match has it own histogram and the match score after 400 games (half of the match) should be close to the end result, preferable not fluctuate more than 1% as is for instance the case in match 2.1
Note that the percentages on this page are true for ProDeo 1.74 only. While it is assumed other chess engines will produce similar percentages it would interesting to know if that is a true assumption.
| Match |
Engine 1 |
Engine 2 |
Win |
Draw |
Loss |
Perc |
Used time |
| 1.1 |
[PLY=6] |
[PLY=7] |
118 |
178 |
503 |
25.9% |
D6 0:20:14 | 0:20:17 | 0:18:59 | 0:18:02
D7 0:37:08 | 0:37:49 | 0:37:00 | 0:37:04
Histogram |
| 1.2 |
[PLY=6] |
[PLY=8] |
49 |
117 |
634 |
13.4% |
D6 0:17:05 | 0:16:46 | 0:16:42 | 0:15:54
D8 1:09:30 | 1:05:02 | 1:06:17 | 1:03:38
Histogram |
| 1.3 |
[PLY=6] |
[PLY=9] |
16 |
88 |
695 |
7.5% |
D6 0:13:44 | 0:14:48 | 0:15:20 | 0:14:02
D9 2:02:32 | 2:04:34 | 2:05:53 | 2:01:40
Histogram |
Comparing the yellow (d+1), blue (d+2) and green (d+3) percentages we can see the diminishing return effect in action, a one ply deeper search (slowly) loses more and more elo gain. An ovierview is given in chapter 2.
| Match |
Engine 1 |
Engine 2 |
Win |
Draw |
Loss |
Perc |
Used time |
| 2.1 |
[PLY=7] |
[PLY=8] |
135 |
207 |
458 |
29.8% |
D07 0:39:38 | 0:38:23 | 0:36:01 | 0:36:28
D08 1:17:56 | 1:17:17 | 1:15:47 | 1:10:54
Histogram 1.2% fluctuation |
| 2.2 |
[PLY=7] |
[PLY=9] |
51 |
149 |
599 |
15.7% |
D07 0:33:57 | 0:32:52 | 0:33:42 | 0:31:36
D09 2:24:53 | 2:20:04 | 2:24:45 | 2:17:56
Histogram |
| 2.3 |
[PLY=7] |
[PLY=10] |
16 |
97 |
687 |
8.1.% |
D07 0:30:12 | 0:30:36 | 0:30:01 | 0:29:39
D10 4:19:40 | 4:27:15 | 4:21:37 | 4:30:44
Histogram |
| Match |
Engine 1 |
Engine 2 |
Win |
Draw |
Loss |
Perc |
Used time |
|
3.1
|
[PLY=8]
|
[PLY=9] |
125 |
230 |
455 |
29.5% |
D8 1:23:33 | 1:17:13 | 1:23:23 | 1:18:43
D9 3:01:35 | 2:46:46 | 3:00:55 | 2:52:24
Histogram |
| 3.2 |
[PLY=8] |
[PLY=10] |
49 |
192 |
560 |
18.1% |
D08 1:12:33 | 1:14:11 | 1:16:22 | 1:13:58
D10 5:21:51 | 5:34:44 | 5:29:27 | 5:39:15
Histogram |
| 3.3 |
[PLY=8] |
[PLY=11] |
13 |
107 |
80 |
8.3% |
D08 1:07:38 | 1:10:40 | 1:11:39 | 1:00:49
D11 10:17:56 | 10:12:46 |10:16:07 | 8:44:13
Histogram |
| Match |
Engine 1 |
Engine 2 |
Win |
Draw |
Loss |
Perc |
Used time |
| 4.1 |
[PLY=9] |
[PLY=10] |
133 |
247
|
420 |
32.0% |
D09 2:52:04 | 2:59:21 | 3:03:29 | 2:53:36
D10 5:57:30 | 6:22:34 | 6:13:27 | 6:03:28
Histogram 1.5% fluctuation |
| 4.2 |
[PLY=9] |
[PLY=11] |
44 |
203 |
558 |
18.1% |
D09 2:33:51 | 3:14:20 | 2:42:33 | 2:34:43
D11 15:37:45 | info loss | 15:25:29 | 15:28:09
Histogram 1.5% fluctuation |
From here we stop our efforts running running depth+2 and depth+3 matches as:
- The elo impact is crystal clear;
- The diminishing return is demonstrated although with 1.5% fluctuation in 4.1 and 4.2 and 1.2% in 2.1 being an indicator that too few games are played for a more exact result.
- These matches take a lot of time.
Instead we prefer to focus on the more realistic (yellow) depth+1 matches at deeper plies. As one can see from the above durations of the matches ProDeo has about a branch factor of +/- 2 which makes the comparison depth+1 = double hardware speed quite acceptable. And so more or less we are also testing the ancient question: what elo do I gain when my PC is 2 x faster?
| Match |
Engine 1 |
Engine 2 |
Win |
Draw |
Loss |
Perc |
Used time |
| 5.1 |
[PLY=10] |
[PLY=11] |
122 |
274
|
406 |
32.3% |
D10 6:19:09 | 6:11:51 | 6:12:35 | 5:59:13
D11 13:11:33 | 12:55:20 | 13:06:42 | 12:27:17
Histogram |
| 6.1 |
[PLY=11] |
[PLY=12] |
125 |
286 |
389 |
33.5% |
D11 19:22:38 | 20:07:50 | 19:54:04 | 20:13:14
D12 40:27:30 | 42:52:19 | 44:08:46 | 43:28:18
Histogram |
| 7.1 |
[PLY=12] |
[PLY=13] |
|
|
|
|
C A N C E L L E D
because it would take 20 days to complete |
Diminishing return overview
chapter 2
DEPTH+1
| Match |
Engine 1 |
Engine 2 |
Gain |
Rough Elo gain |
Estimated time control |
| 1.1 |
[PLY=6] |
[PLY=7] |
50 - 25.9 = 24.1% |
180 |
10s/all | 20s/all |
| 2.1 |
[PLY=7] |
[PLY=8] |
50 - 29.8 = 20.2% |
147 |
20s/all | 40s/all |
| 3.1 |
[PLY=8] |
[PLY=9] |
50 - 29.5 = 20.5% |
151 |
40s/all | 40/0:40 |
| 4.1 |
[PLY=9] |
[PLY=10] |
50 - 32.0 = 18.0% |
129 |
40/0:40 | 40/1:20 |
| 5.1 |
[PLY=10] |
[PLY=11] |
50 - 32.3 = 17.7% |
127 |
40/1:20 | 40/2:30 |
| 6.1 |
[PLY=11] |
[PLY=12] |
50 - 33.5 = 16.5% |
117 |
40/2:30 | 40/5m |
As we can see the elo gain by (roughly) doubling the speed of the computer remains remarkbale, 117 elo.
DEPTH+2
| Match |
Engine 1 |
Engine 2 |
Gain |
Rough Elo gain |
Estimated time control |
| 1.2 |
[PLY=6] |
[PLY=8] |
50 - 13.4 = 37.6% |
321 |
10s/all | 40s/all |
| 2.2 |
[PLY=7] |
[PLY=9] |
50 - 15.7 = 34.3% |
281 |
20s/all | 40/0:40 |
| 3.2 |
[PLY=8] |
[PLY=10] |
50 - 18.1 = 31.9% |
255 |
40s/all | 40/1:20 |
| 4.2 |
[PLY=9] |
[PLY=11] |
50 - 18.1 = 31.9% |
255 |
40/0:40 | 40/2:30 |
DEPTH+3
| Match |
Engine 1 |
Engine 2 |
Gain |
Rough Elo gain |
Estimated time control |
| 1.3 |
[PLY=6] |
[PLY=9] |
50 - 7.5 = 42.5% |
401 |
10s/all | 40/0:40 |
| 2.3 |
[PLY=7] |
[PLY=10] |
50 - 8.1 = 41.9% |
389 |
20s/all | 40/1:20 |
| 3.3 |
[PLY=8] |
[PLY=11] |
50 - 8.3 = 41.7% |
386 |
40s/all | 40/2:30 |
Elo gain is calculated via the table:
| Percentage |
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
| 01 - 10 |
4 |
11 |
18 |
26 |
33 |
40 |
47 |
54 |
62 |
69 |
| 11 - 20 |
76 |
83 |
91 |
96 |
106 |
113 |
121 |
129 |
137 |
145 |
| 21 - 30 |
153 |
162 |
170 |
179 |
188 |
197 |
206 |
215 |
225 |
235 |
| 31 - 40 |
245 |
256 |
267 |
278 |
290 |
302 |
316 |
328 |
344 |
357 |
| 41 - 50 |
374 |
391 |
411 |
432 |
456 |
484 |
517 |
559 |
619 |
735 |
Diminishing returns the classic way
chapter 3
Since Match 7.1 (see above) is undoable (800 games would take 20 days to complete) we take a faster ProDeo version (1.81) and repeat the experiment but also now without the [PLY = x] formula. Meaning standard PLY=x vs PLY=x+1 matches. This will speedup the matches considerable with as goal to see the diminishing return effect at deeper depths.
The disadvantage of this procedure is that the focus of the games will be on the middle game as (for instance) 14 ply in the middle game is something else as 14 ply in the end game. Nevertheless it will be interesting to notice if the figures will match the above ones.
Using ProDeo 1.81
| Match |
Engine 1 |
Engine 2 |
Score
ProDeo 1.81 |
Elo gain
ProDeo 1.81 |
Elo gain
ProDeo 1.74 |
Total used time |
Games |
| 1.1 |
[PLY=10] |
[PLY=11] |
50 - 31.8 = 18.2% |
131 |
127 |
D10 15:49:49
D11 32:19:23 |
800 |
| 1.2 |
[PLY=11] |
[PLY=12] |
50 - 30.8 = 19.2% |
139 |
117 |
D11 38:27:19
D12 79:56:12 |
800 |
| 1.3 |
[PLY=12] |
[PLY=13] |
50 - 32.8 = 17.2% |
123 |
--- |
D12 38:38;45
D13 78:44:34 |
400 |
| 1.4 |
[PLY=13] |
[PLY=14] |
50 - 34.0 = 16.0% |
113 |
--- |
D13 44:30:37
D14 92:48:01 |
213 |
| 1.5 |
[PLY=14] |
[PLY=15] |
50 - 36.6 = 13.4% |
93 |
--- |
D14 64:12:24
D15 127:50:03 |
101 |
As we can see the elo gain by (roughly) doubling the speed of the computer remains remarkbale, bordering around +/- 100 elo even at such deep depths.
This ends our study with the satisfying thought that one day ProDeo will cross the 3000 elo barrier  | 
Quick downloads
