Wisdom is knowing how

   little we know -- Socrates

About Company

Started in 1980, retired in 2004 REBEL was baptized into ProDeo, latin for gratis according to Dutch tradition.

Other Information

    Quick downloads

 

 

    Quick links

 

 

    Technical

 

 

    Misc

 

 

    Personal links

 

 

 

Experiments in computer chess

 

The value of depth and diminishing return effects.

 

For this purpose we took ProDeo 1.74 playing depth based matches of (only) 800 games (4 x 200) on a Quad PC using ProDeo's parameter:

 

[PLY=x]

 x is the base depth for the middle game phase and gradually increased when the game

 moves into different phases.

 

When for example using [PLY = 8] then:

 

  1. All middle game positions will be played at depth 8.
  2. When Queens are off the base depth (8) is raised by 1 and becomes 9.
  3. Normal endgames will be played with depth 10.
  4. Simple endgames (rook endings) the base search depth increases with 3 and becomes 11.
  5. Light piece endings are searched with +4 thus 12 plies deep.
  6. And finally pawn endings at 13 plies.
  7. We make one exception for Queen endings, the depth increase is only 1, thus 9 plies because of the many check extensions that usually play a role in Queen endings.

   

This all to ensure that games based on depth will relative closely emulate the same depth behavior as playing games on time control. The system has the dvantage that it avoids any intereference from outside. Contrary to time control games depth based games always end the same way while time control based games are dependent on the capriciousness of the operating system (Windows and friends) and user interferences doing things into the background. As such we are excluding randomness in a process that is already so sensitive to randomness.

 

The error margin of 800 games is quite high but since we are dealing with big elo differences it's accetable to allow fluctuations of 1-2% max. For that purpose each match has it own histogram and the match score after 400 games (half of the match) should be close to the end result, preferable not fluctuate more than 1% as is for instance the case in match 2.1

 

Note that the percentages on this page are true for ProDeo 1.74 only. While it is assumed other chess engines will produce similar percentages it would interesting to know if that is a true assumption.

 

 Match  Engine 1  Engine 2  Win Draw Loss  Perc  Used time
 1.1  [PLY=6]  [PLY=7]  118  178  503  25.9% 

 D6  0:20:14 | 0:20:17 | 0:18:59 | 0:18:02

 D7  0:37:08 | 0:37:49 | 0:37:00 | 0:37:04

 Histogram

 1.2  [PLY=6]  [PLY=8]   49  117  634  13.4%

 D6  0:17:05 | 0:16:46 | 0:16:42 | 0:15:54

 D8  1:09:30 | 1:05:02 | 1:06:17 | 1:03:38

 Histogram

 1.3  [PLY=6]  [PLY=9]   16    88  695     7.5%

 D6  0:13:44 | 0:14:48 | 0:15:20 | 0:14:02

 D9  2:02:32 | 2:04:34 | 2:05:53 | 2:01:40

 Histogram

 

Comparing the yellow (d+1), blue (d+2) and green (d+3) percentages we can see the diminishing return effect in action, a one ply deeper search (slowly) loses more and more elo gain. An ovierview is given in chapter 2.

 

 Match  Engine 1  Engine 2  Win Draw Loss  Perc  Used time
 2.1  [PLY=7]  [PLY=8]  135  207   458  29.8%

 D07  0:39:38 | 0:38:23 | 0:36:01 | 0:36:28

 D08  1:17:56 | 1:17:17 | 1:15:47 | 1:10:54

 Histogram      1.2% fluctuation

 2.2  [PLY=7]  [PLY=9]   51  149   599   15.7%

 D07  0:33:57 | 0:32:52 | 0:33:42 | 0:31:36

 D09  2:24:53 | 2:20:04 | 2:24:45 | 2:17:56

 Histogram

 2.3  [PLY=7]  [PLY=10]   16   97  687  8.1.%

 D07  0:30:12 | 0:30:36 | 0:30:01 | 0:29:39

 D10  4:19:40 | 4:27:15 | 4:21:37 | 4:30:44

 Histogram

 

 Match  Engine 1  Engine 2  Win Draw Loss  Perc  Used time

 3.1

 

 [PLY=8]

 

 [PLY=9]  125  230  455  29.5%

 D8   1:23:33 | 1:17:13 | 1:23:23 | 1:18:43

 D9   3:01:35 | 2:46:46 | 3:00:55 | 2:52:24

 Histogram

 3.2  [PLY=8]  [PLY=10]   49  192  560  18.1%

 D08  1:12:33 | 1:14:11 | 1:16:22 | 1:13:58

 D10  5:21:51 | 5:34:44 | 5:29:27 | 5:39:15

 Histogram 

 3.3  [PLY=8]  [PLY=11]  13  107  80  8.3%

 D08    1:07:38 |  1:10:40 |  1:11:39 | 1:00:49

 D11  10:17:56 | 10:12:46 |10:16:07 | 8:44:13

 Histogram

 

 Match  Engine 1  Engine 2  Win Draw Loss  Perc  Used time
 4.1  [PLY=9]  [PLY=10]  133

 247

 

 420  32.0%

 D09  2:52:04 | 2:59:21 | 3:03:29 | 2:53:36

 D10  5:57:30 | 6:22:34 | 6:13:27 | 6:03:28

 Histogram        1.5% fluctuation

 4.2  [PLY=9]  [PLY=11]   44  203 

558

 18.1%

 D09   2:33:51 | 3:14:20 |   2:42:33 |   2:34:43

 D11 15:37:45 | info loss | 15:25:29 | 15:28:09

 Histogram        1.5% fluctuation

 

From here we stop our efforts running running depth+2 and depth+3 matches as:

 

  1. The elo impact is crystal clear;
  2. The diminishing return is demonstrated although with 1.5% fluctuation in 4.1 and 4.2 and 1.2% in 2.1 being an indicator that too few games are played for a more exact result.
  3. These matches take a lot of time.

 

Instead we prefer to focus on the more realistic (yellow) depth+1 matches at deeper plies. As one can see from the above durations of the matches ProDeo has about a branch factor of +/- 2 which makes the comparison depth+1 = double hardware speed quite acceptable. And so more or less we are also testing the ancient question: what elo do I gain when my PC is 2 x faster?

 

 Match  Engine 1  Engine 2  Win Draw Loss  Perc  Used time
 5.1  [PLY=10]  [PLY=11]  122

 274

 406  32.3%

 D10    6:19:09 |   6:11:51 |  6:12:35 |   5:59:13

 D11  13:11:33 | 12:55:20 | 13:06:42 | 12:27:17

 Histogram

 6.1  [PLY=11]  [PLY=12]  125   286  389   33.5%

 D11  19:22:38 | 20:07:50 | 19:54:04 | 20:13:14

 D12  40:27:30 | 42:52:19 | 44:08:46 | 43:28:18

 Histogram

 7.1  [PLY=12]  [PLY=13]            

C A N C E L L E D

because it would take 20 days to complete

 

 

Diminishing return overview

chapter 2

 

DEPTH+1

 Match  Engine 1  Engine 2  Gain  Rough Elo gain  Estimated time control
 1.1  [PLY=6]  [PLY=7]  50 - 25.9 = 24.1%

180

 10s/all  | 20s/all
 2.1  [PLY=7]  [PLY=8]  50 - 29.8 = 20.2%

147

 20s/all  | 40s/all
 3.1  [PLY=8]  [PLY=9]  50 - 29.5 = 20.5%

151

 40s/all  | 40/0:40
 4.1  [PLY=9]  [PLY=10]  50 - 32.0 = 18.0%

129

 40/0:40 | 40/1:20
 5.1  [PLY=10]  [PLY=11]  50 - 32.3 = 17.7%

127

 40/1:20 | 40/2:30
 6.1  [PLY=11]  [PLY=12]  50 - 33.5 = 16.5%

117

 40/2:30 | 40/5m

 

As we can see the elo gain by (roughly) doubling the speed of the computer remains remarkbale, 117 elo.

 

DEPTH+2

 Match  Engine 1  Engine 2  Gain  Rough Elo gain  Estimated time control
 1.2  [PLY=6]  [PLY=8]  50 - 13.4 = 37.6%

321

 10s/all  | 40s/all
 2.2  [PLY=7]  [PLY=9]  50 - 15.7 = 34.3%

281

 20s/all  | 40/0:40
 3.2  [PLY=8]  [PLY=10]  50 - 18.1 = 31.9%

255

 40s/all  | 40/1:20
 4.2  [PLY=9]  [PLY=11]  50 - 18.1 = 31.9%

255

 40/0:40 | 40/2:30

 

DEPTH+3

 Match  Engine 1  Engine 2  Gain  Rough Elo gain  Estimated time control
 1.3  [PLY=6]  [PLY=9]  50 - 7.5 = 42.5%

401

 10s/all  | 40/0:40

 2.3  [PLY=7]  [PLY=10]  50 - 8.1 = 41.9%

389

 20s/all  | 40/1:20
 3.3  [PLY=8]  [PLY=11]  50 - 8.3 = 41.7%

386

 40s/all  | 40/2:30

 

 

Elo gain is calculated via the table:

 

 Percentage    1    2     3    4    5     6    7     8     9   10
 01 - 10    4   11   18   26   33    40   47   54   62   69
 11 - 20   76   83   91   96  106  113  121  129  137  145
 21 - 30  153  162  170  179  188  197  206  215  225  235
 31 - 40  245  256  267  278  290  302  316  328  344  357
 41 - 50  374  391  411  432   456  484  517  559  619  735

 

 


 

Diminishing returns the classic way

chapter 3

 

Since Match 7.1 (see above) is undoable (800 games would take 20 days to complete) we take a faster ProDeo version (1.81) and repeat the experiment but also now without the [PLY = x] formula. Meaning standard PLY=x vs PLY=x+1 matches. This will speedup the matches considerable with as goal to see the diminishing return effect at deeper depths.

 

The disadvantage of this procedure is that the focus of the games will be on the middle game as (for instance) 14 ply in the middle game is something else as 14 ply in the end game. Nevertheless it will be interesting to notice if the figures will match the above ones.

 

Using ProDeo 1.81

 

 Match  Engine 1  Engine 2

Score

 ProDeo 1.81

Elo gain

 ProDeo 1.81

Elo gain

 ProDeo 1.74

 Total used time

 Games

 1.1  [PLY=10]  [PLY=11]  50 - 31.8 = 18.2%

131

 127

 D10    15:49:49

 D11    32:19:23

800

 1.2  [PLY=11]  [PLY=12]  50 - 30.8 = 19.2%

139

 117

 D11    38:27:19

 D12    79:56:12

800

 1.3  [PLY=12]  [PLY=13]

50 - 32.8 = 17.2%

 123

 ---

 D12    38:38;45

 D13    78:44:34

400

 1.4  [PLY=13]  [PLY=14]

 50 - 34.0 = 16.0%

 113

 ---

 D13    44:30:37

 D14    92:48:01

213

 1.5  [PLY=14]  [PLY=15]  50 - 36.6 = 13.4%

 93

 ---

 D14    64:12:24

 D15  127:50:03 

101

 

As we can see the elo gain by (roughly) doubling the speed of the computer remains remarkbale, bordering around +/- 100 elo even at such deep depths.

 

This ends our study with the satisfying thought that one day ProDeo will cross the 3000 elo barrier

Copyright ® 2012  Ed Schröder