The Rybka Investigation:
A Survey Of Evidence From Both Perspectives
by Adam Hair
The purpose of this exposition (which, given its intended neutral position, is most likely to be quickly forgotten) is to collect, into one place, all evidence presented in this debate by both sides. The debate has been conducted across at least 3 forums, with the evidence/counterevidence spread among these forums, the ICGA investigation wiki, the ChessBase website, Ed Schröder's website, and Mark Watkins' PDF reports.
To counteract this dispersion, I am attempting (perhaps futilely) to bring focus to the debate and to provide a commentary for the general observer, while remaining neutral. None of the commentary on
the investigation and subsequent debate has provided a neutral point of view, either out of choice/intent (such as Søren Riis' 4 part rebuttal for ChessBase) or due to the nature of the commentary (such as Mark Watkins' statements concerning the investigation, given his role in the investigation).
Why try to provide a neutral commentary? Well, in my opinion the outcome of this debate is very important to the general computer chess community, regardless if the resulting consensus is that Vas Rajlich is innocent of the charges or guilty. Of course, for certain factions it is of little importance if Vas did bend or break rules pertaining to ICGA competitions. Since the infamous "rule #2" is a rule determined by and enforced by the ICGA solely for its tournaments, I agree that it is of no importance outside of those tournaments.
The true importance is that an investigation was made of Rybka and the investigators determined, in their opinion, that Vas plagiarized the work of Fabian Letouzey. Whether or not their opinion should become the consensus of the community should be the heart of this debate, in my opinion. For some members of this community, this does not matter. Of upmost importance to them is the increase in strength (either for play or analysis) of the top chess engines. How the increase occurs matters little, if at all. For some others, they are fans of Vas and Rybka.
Perhaps they are members of the Rybka community. To many of those people, allegations of plagiarism are seen as an attack against their community. Then, there is the other end of the spectrum. People who might be fans of other engines. Also, people who dislike Vas and Rybka simply due to the fact that Rybka was on top for a long time. Anyone who has made some note of human nature would have to believe that some of those people exist. Then there are people who may have taken one side because some other person took the other side.
Finally, there are people who have decided the guilt or innocence of Vas to such a degree that they do not give serious consideration to contrary opinions. All of the people described in the preceeding paragraph have a vested interest in the outcome of this debate. Unfortunately, that interest is personal and is not connected to the well-being of the computer chess community at large. For the majority of those people, their input to the debate has been noise. For the minority of those people whose signal to noise ratio was greater than 1, there is bias present. Their statements have to be taken with a grain of salt.
It is important to determine, as a community, if Vas committed a breach of ethics or not. A determination in this case provides guidance as to what is considered acceptable behavior. Obviously, there is not a consensus at this time. Most important is to acheive a consensus among the engine authors, but ideally among the enthusiasts also. To achieve this, it is important to neutralize the noise and bias. Presenting both sides of the debate in a neutral manner is a step in that direction. To the best of my ability, I am going to present all evidence and counterevidence and the relevant discussion pertaining it all.
I will use Søren Riis' ChessBase article, Ed Schröder's website dedicated to examining the investigation, and Mark Watkin's RECAP pdf as well as "A Critical Analysis of The Four Parts of Riis" as my primary sources for the evidence/counterevidence, as well as posts from the major fora. For some of this, my lack of technical knowledge will by force limit me to simply recapitulate the points under debate. For some points, I can comment with some intelligence and will do so (noted by the use of a * symbol preceeding my comments). I will try to act as a devil's advocate towards both sides. I do not claim to be completely neutral, but I am not married to my opinion. My desire is that a consensus is achieved, not on what that consensus is. The reader can judge whether I have maintained neutrality or not. I hope that the resulting work is worth the time spent reading it.
First, a short discussion of what will not be covered.
1) The ICGA and rule #2
The ICGA's investigation was to determine if versions of Rybka that competed in ICGA tournaments were in fact in violation of its rule #2. The clear fact is that it can determine its own rules and enforce them as it sees fit. While the spirit of rule #2 is of great importance to the general community, a debate of the ICGA's relevance and of the wording of rule #2 and its enforcement by the ICGA is of lesser importance to the general community.
2) Vas Rajlich's contributions to computer chess
While not universally agreed upon, Vas' contributions are seen by some as considerable. However, a discussion of his contributions would be more relevant in regards to determining how he should be treated by the general community if the general opinion is that he is guilty of plagiarism, not in determining if he is innocent or guilty.
Note: Following Ed Schröder's notation, I will use VIG (Vas is Guilty) to label the views of the group who feel that Vas is guilty and VII(Vas is Innocent) to label the competing point of view.
Pre-Rybka 1.0 Beta versions and Crafty 19
VIG: Versions of Rybka that were submitted to Oliver Deville (operator of the computer chess tournament 'ChessWar') in 2004 were found, via reverse engineering, to have substantial portions of Crafty 19 code, related to both the search and evaluation. This example of code-copying exhibits that Vas was willing to use the code of other authors in a manner contrary to the original author's wishes.
source: Mark Watkins' reports, comments (most importantly, Robert Hyatt's comments on various fora)
VII: The veracity of the discovery has not been questioned. The relevance of the discovery, epecially the relevance to the connection between Fruit 2.1 and Rybka 1.0 Beta, has been questioned. This is not an uncommon occurance among new authors (using portions of open source code as place-holders for code they are yet to write). Since the versions in question were quite weak in comparison to Rybka 1.0 Beta (and also Crafty 19), the crime is minor. The true intent of these allegations is to pre-condition the reader into believing Vas is a code-copier.source: Rybka forum
of UCI Parameters
Order and Existence
VII: To quote Ed Schröder:
"In a situation that you as a programmer have to check 20-30 items such as in UCI with no demanding order then what do you pick first? It will depend from programmer to programmer as the nature is completely random. As such taking UCI as an example every programmer that does not "copy" will leave his own signature in the ORDER of programming UCI.
As such the order of handling things in a random process can be seen as evidence for copying. The opposite is also true, the lack of an equal or very similar order should count as unburden evidence for copying.
The facts: the order of UCI in FRUIT is totally different than order of UCI in RYBKA. In fact most of the engine features in Fruit are missing in Rybka 1 best demonstrated by a visualization. (Note: The visualization consists of screenshots of UCI parameters for Fruit 2.1 and Rybka 1.0 Beta using the Fritz GUI)
And so we are back at obfuscation square one, Vasik in 2005 already foresaw he became so famous some people would disassemble his program looking for Fruit traces and therefore Vasik deliberately decided to obfuscate the UCI protocol by removing nice EVAL and SEARCH settings and replace it with a new one for maximal obfuscation. This is a bizarre scenario.
Every open chess source I have checked has a complete different order in UCI and its own features because of its random nature, my own included and as one might expect FRUIT and RYBKA are no exceptions. They all have a different order in UCI as it should.
Both the contents and order of Rybka’s UCI parameters are different from those of Fruit (just like we saw with the Rybka and Fruit evaluations. Similar order in a random process is evidence for copying, and the opposite is an indication that Rajlich wrote his own UCI parser code. Fruit 2.1 has twenty configurable UCI parameters, this is what the majority of them appear as in the Chessbase Fritz GUI (note that hashsize does not appear in this dialog)."
VIG: It is implied that comparison of UCI parameters is an insufficient standard for determining the relationship between engines. In particular, Rybka and Ippolit would be considered unrelated (in contrast to the insinuation made in the Søren Riis article).
* Using this as a way to judge whether two engines were related or not would lead one to believe that Doch 09.980 and Komodo 1.0 are unrelated engines. In fact, the author of Doch is Don Dailey. He decided to change the name of his engine, and asked for input from members of TalkChess. Komodo was the winning submission.
Order of Components in Rybka 1.0 Beta's Eval() as compared to Fruit 2.1
VII: Much like the UCI parameters, the order of the components of the Eval function of a chess engine do not have to be inserted in a particular order. It depends on the whims of the author. In a case of code copying, one would expect that the order of the components in the two engines would be close to the same. But the order in Fruit 2.1 and Rybka 1.0 Beta is not the same.
VIG: To quote Zach Wegner:
I really don't see evaluation order as being that relevant at all, certainly not to the extent you are claiming. But as Mark showed, the evaluation order is in fact quite similar, pretty much exactly what I would expect from a bitboard translation (except the weird placement of eval_pawns()).
*In the same thread at the Rybka forum, Chris Whittington and Zach Wegner both implied that the order of Eval() is of little importance in determining guilt or innocence. With some effort, a plagiarist could change the order of the components. On the other hand, a person that is using some of the ideas of Fruit might innocently use the same arrangement of the individual components.
0.0 in time management
The floating-point number
VIG: Rybka 1.0 Beta uses an interger-based time control and Fruit 2.1 uses a floating point (rational number) based time control. However, the floating point number 0.0 appears in Rybka's time control code, along with other similarities to Fruit 2.1. While there are some differences, certain characteristics of Fruit's UCI parsing and the mixture of that parsing code with time management code is echoed in Rybka 1.0 Beta. Curiously, Rybka 1.0 Beta and Rybka 1.6.1 due not share these characteristics.
source: RECAP.pdf (can be downloaded here: http://www.open-chess.org/viewtopic.php?f=5&t=1772)
VII: It is possible that Vas typed 0. instead of 0 (accidental insertion of a period). A compiler could interpret 0. as 0.0. Furthermore, the code surrounding the 0.0 appears to be dissimilar to Fruit. Compare the Fruit 2.1 code with Rick Fadden reconstucted Rybka C code
( http://www.talkchess.com/forum/viewtopic.php?p=187290#187290 ) and note the differences
("immense" according to Ed Schröder).
*I do not possess the requisite knowledge to compare and contrast Fruit code and Rybka asm or reconstucted Rybka code. I must note that, in order to judge Rick Fadden's reconstructed code, I read the two responses to his post where he is comparing Strelka 2.0 to Rybka by analyzing the decompiled code from Rybka 1.0 Beta and then interpreting the asm in order to construct a C code representation of the asm. Fonzy Bluemers and Norman Schmidt both noted that the resulting code resembled code found in protocol.cpp of Fruit 2.1's source code. I also must note that Rick's response to Fonzy:
looks almost exactly like the fruit/toga code to me
protocol.cpp if i remember well
strelka using integers instead of (slower) double
Rick Fadden: Yes, let's check into that, but note that I have matched up the thousands of lines that follow and they are all practically identical between Rybka and Strelka, and this is the one spot where I am showing some slight additions being made to Strelka (start_go).
What I am trying to say is that the next piece of work that I am posting shows Strelka as a pure reverse enginer of Rybka, and these two identical programs are not like Fruit. These two programs have the same constants, Strelka was derived from Rybka, and you do not see these constants or anything like this in Fruit.
I wanted to go through step 1 to get to step 2, but I have already seen the answer in steps 2 through 279 and they all match so all I am doing is bringing out the material that I have seen in great detail, so others can also see this.
Finally, I must note that Fabian Letouzey implied in his open letter to the community (
http://talkchess.com/forum/viewtopic.php?t=37762) that Strelka 2.0 was similar to Fruit 2.1.
Search control code in Fruit 2.1 and Rybka 1.0 Beta
VIG: This sort of code determines what the engine should do once the search has concluded. Mark Watkins asserts the code is "quite idiosyncratic to Fruit 2.1 (it differs slightly in Fruit 1.0), and re-appears essentially verbatim in Rybka 1.0 Beta."source: RECAP.pdf http://www.open-chess.org/viewtopic.php?f=5&t=1772
VII: I have found no response to this.
* Mark Watkins did note that it is a rather short piece of code. It is up to the reader to determine if the importance of a piece of code is determined by its length or if it is determined by its function (importance in the context of code copying).
Control of iterative deepening
VIG: The claim made is much like that for search control code. There seems to be some unique aspects to Fruit 2.1's code that appears in Rybka 1.0 Beta. source: RECAP.pdf http://www.open-chess.org/viewtopic.php?f=5&t=1772
VII: I have found no response to this.
The SETJMP case
VII: The use of setjmp() in Fruit 2.1 and Rybka 1.0 Beta is not unusual. Indeed, other programmers have and still do use it.
VIG: In chapter 4 of Mark Watkins' Rybka_Fruit_Mar11.pdf, he lists side-by-side the order of root search procedures for both Fruit 2.1 and Rybka 1.0 Beta. setjmp() is listed in the procedures for both programs. The point of chapter 4 appears to be that the same procedures are found in the same order. setjmp() itself is not noted to be of individual importance.source: Rybka_Fruit_Mar11.pdf (can be found here:
* The commentary on the use of setjmp() appears to originate with this thread at OpenChess forum (see page 5):
Mark Watkins (BB+) throws in setjmp() as a possible suspicious similarity among other
things in the root search of Fruit 2.1 and Rybka 1.0 Beta (
I think the person who placed the most importance on this particular procedure is Robert Hyatt. Chris
Whittington attributed this quote to Dr. Hyatt:
The first one that jumped out at me when we started the process was the code segment containing the setjmp()/longjmp() construct. You can find references to this in the past. It is an unusual way to unwind a search that is not thread-happy, and invites very subtle bugs. The usage was identical in both programs with the surrounding code. There were others.
Given the date of the post from which the quote is taken ( July 01, 2010 ), the "process" must refer to the work done by Zach Wegner to decompile Rybka 1.0 Beta and interpret the resulting asm. In the official ICGA investigation, no individual significance is placed on setjmp(). Indeed, given that other programmers do use setjmp(), its individual importance in proving guilt is nil. Thus, only its position in the order of the root search seems to hold any importance in the investigative report.
Differences between Fruit 2.1 and Rybka 1.0 Beta
VII: Ed Schröder lists these differences between Fruit 2.1 and Rybka 1.0 Beta
- Mailbox vs Bitboard
- History Reductions vs LMR
- Different Futility Pruning
- Rybka 1.0 Beta uses Lazy Evaluation
- The absence of the quad function in Fruit 2.1
- Material Imbalance Table in Rybka 1.0 Beta
- Rybka 1.0 Beta lacks minor promotions
VIG: These are noted in the investigation.
* It has been stated that several of these differences can be looked upon as improvements on Fruit 2.1. However, that statement alone contributes nothing to the determination of guilt. I am not sure how much weight this contributes to a determination of innocence, but it must contribute.
VIG: Mark Watkins conducted a pairwise comparision of evaluation features found in a selection of open source engines that existed in 2005 and also Rybka 1.0 Beta and Rybka 2.3.2a. For each feature and each pair of engines, he determined a measurement of "feature overlap" present. He then used standard statistical procedures to determine the significance of the measure of "feature overlap" for each pair of engines. He found that the Rybka/Fruit overlap was significantly larger than for other pairs.
VII: The most significant challenges to this assessments can be found in Søren Riis' ChessBase article.
- The "feature overlap" is not "code overlap"
- The engines included in addition to Fruit and Rybka are relatively weak in comparison to them.
- The assessment of the "feature overlap" was subjective. If a group of independent investigators were asked to perform similar measurements, it is not known how close his measurements would be to the average.
- What makes a 44% overlap permissible but a 74% overlap (such as the case with Fruit 2.1 and Rybka1.0 Beta) impermissible?
- *An evaluation of the first point is non-technical and any reader can do the requisite reading to make their own judgement. Watkins presents the point of view that a large overlap of features (ideas) could represent a lack of originality. This seems to be very applicable to the ICGA case. For me, I wonder if such an overlap is something that the general community wishes to discourage, especially for closed source engines. An overlap of code is a definite breach of ethics. Is a large overlap of ideas also a breach?
- * The engines in the comparison are indeed weaker than Fruit and Rybka. Nobody with the required knowledge has stated that these engines are weaker primarily due to a lack of evaluation knowledge as compared to Fruit and Rybka. Presumably Faile, Resp, and EXchess
lack enough knowledge. But it is not clear that Crafty, Phalanx, and Pepito are so deficient in regards to evaluation such that comparisons to Fruit and Rybka are suspect.
Vas himself is said to have stated (part 3 of Dr. Riis' ChessBase article) that Fabian was
a "big search guy". Presumably, differences in search plays some role in the difference in
strength between Fruit and those other engines.
- * It is entirely possible that bias was present in Mark Watkins' overlap assessments. After all, he is a human being. Therefore, it is possible that his measurements would be at variance to an average of measurements made by others. I do think it could be agreed that his possible bias is not to an extreme degree.
Perhaps he overvalued the Rybka/Fruit overlap by 10% and undervalued all the other overlaps by 10%. If we were to compare, pairwise, each of the open source engines as well as Rybka 1.0 Beta using the revised overlaps, the new overlap for Fruit 2.1/Rybka 1.0 Beta would be 67 and that would be ~3.76 standard deviations from the mean.
This represents 1 in 11,771 chance of occurring naturally. Of course, if there was more bias, then Mark Watkins' measurements would be even more inaccurate and the significance of the Fruit 2.1/Rybka 1.0 Beta would lessen even more.
- * If the basis for the measurements has validity, then standard statistical procedures determine how to judge the significance of the measurements. If a data point is considered to be a significant outlier, then it is justified in questioning if it is a natural occurrence or not.
Piece Square Tables (PSTs)
VIG: Rybka 1.0 Beta's PST values can be reproduced with an abnormally small number of
changes to Fruit 2.1's code.
source: RECAP.pdf http://www.open-chess.org/viewtopic.php?f=5&t=1772
VII: The Fruit 2.1 code that produces the PST values has low information entropy (sort of a measure of information content) and the bit of information contained is general chess knowledge. In addition, code different from Fruit 2.1's code can reproduce the values.
source: Rybka and OpenChess forums
* Can the PSTs of any pre-Fruit 2.1 engine be reproduced as simply as Rybka 1.0 Beta's? Does it really matter? I am not certain.
Playing similarity testing of computer
VII: Analysis of similarity test data with clustering algorithmns shows that Fruit 2.1 and
Rybka 1.0 Beta are unrelated.
VIG: Similarity data, while interesting, is an inconclusive tool for determining originality.
* Let me go into some detail here. There are several closed source engines that appear to be more closely related (in terms of move selection) to Fruit 2.1 than Rybka 1.0 Beta. Indeed, clustering programs place some of these engines in the same branch of the resulting phylogenetic tree as Fruit 2.1, while Rybka 1.0 Beta is placed in a different branch.
However, the Fruit 2.1/Rybka 1.0 Beta pair is still a significant outlier, in terms of how outliers are normally judged, just not nearly as significant as some other pairs. In my opinion, the similarity data neither condemns nor exonerates Vas.
These are the major points that I see in terms of judging the guilt or innocence of Vas Rajlich. There are a couple of things that I ignored, things brought up in the name of VIG or VII.
VIG: Rybka increased 600+ Elo in the 5½ months after the release of Fruit 2.1.
* The truth of the matter is that this does not signify Vas is guilty. It does signify something occurred within Rybka, some breakthroughs, that allowed the increase in strength. Indeed, the investigation uncovered that Rybka 1.0 Beta bears little resemblence to the earlier Rybkas and it appears to have been completely rewritten, at least in terms of the evaluation.
However, we do know that Vas had a revolutionary idea in terms of testing (large number of super-fast games) that speed up testing and confirmation of code changes.
Also, we do not know when the rewrite started. All that we know, confirmed by Vas' own statement, is that Fruit 2.1 played some role in the increase.
VII: If you are 150 Elo above all rest, you have an original engine.
* Rybka 1.0 Beta was not 150 Elo above the competition. Also, even if it was, the notion that the improvements were built upon a solid base (Fruit 2.1) has not been debunked. Fabian himself stated that there was a lot of Elos that could be gained from Fruit. Vas may have done that.
There are many more things that have been brought up during the community-wide debate. Some are minor factual details. Others are more speculative in nature. Also half-truths, exaggerations, libel, insults, outright lies. All in the name of swaying the audience (or just to jab at another person). Rhetorical devices have been used instead of logic.
Propaganda instead factual statements. People seem to be much more concerned about winning the debate than about discovering the truth. The community as a whole has been guilty of this. It is obvious that many are not concerned about the community as a whole, just their part of it.
There are technical aspects that I don't understand. Common programming practices that I know nothing about. If I have misrepresented a position, I apologize for that; it was not intentional. I do hope that I have collected into one spot all of the major points that are worth debating and that it helps members of the general audience get a better feel of what is being debated, without having to wade through the trash talk. Above all, I do hope this attempt is considered neutral.