The British and Americans developed a key device for expanding the VENONA breaks. It was called a "window index." Every time a word or phrase was broken out, it was indexed to everywhere else it appeared in the matched traffic. The British began to index these decrypts in a more advanced way. They placed two unsolved groups on each side of the decrypted word or phrase and after a period of time these window indexes led to repetitions, where different words which had been broken out were followed by the same unsolved group. The repetition often gave enough collateral to begin a successful attack on the group, thus widening the window indexes. Another technique was "dragging." Where a "Spell/Endspell" sequence or name came up, and the cryptanalysts did not know what the missing letters of the spelled sequence were, the groups were dragged, using a computer, across the rest of the channels, and out would come a list of all the repeats. Then the cryptanalysts would set to work on the reverse side of the repeat matches, and hope to attack the "Spell/Endspell" sequence that way.
It was an imperfect art, often moving forward only a word or two a month, and then suddenly spilling forward, like the time the Americans found the complete text of a recorded speech in the Washington Ambassadorial channel. Often terrible new difficulties were encountered: one-time pads were used in unorthodox ways, up and down, or folded, which made the process of finding matches infinitely more problematic. There were difficulties, too, with the codebooks. Sometimes they changed, and whereas the Ambassadorial, GRU, and trade channels used a straightforward alphabetically listed codebook, rather like a dictionary, so that the codebreakers could guess from the group where in the codebook it appeared, the KGB used a special multivolume random codebook which made decrypting matched KGB channels a mindbending task. The effort involved in VENONA was enormous. For years both GCHQ and NSA and MI5 employed teams of researchers scouring the world searching for "collateral"; but despite the effort less than 1 percent of the 200,000 messages we held were ever broken into, and many of those were broken only to the extent of a few words.
But the effect of the VENONA material on British and American intelligence was immense, not just in terms of the counterintelligence received, but in terms of the effect it had on shaping attitudes in the secret world. By the late 1940s enough progress was made in the New York/Moscow and Washington/Moscow KGB channels to reveal the extent of massive Russian espionage activity in the USA throughout and immediately after the war. More than 1200 cryptonyms littered the traffic, which, because they were frequently part of "Spell/Endspell" sequences, were often the easiest things to isolate in the traffic, even if they could not be broken. Of those 1200, more than 800 were assessed as recruited Soviet agents. It is probable that the majority of these were the low-level contacts which are the staple currency of all intelligence networks. But some were of major importance. Fourteen agents appeared to be operating in or close to the OSS (the wartime forerunner of the CIA), five agents had access, to one degree or another, to the White House, including one who, according to the traffic, traveled in Ambassador Averill Harriman's private airplane back from Moscow to the USA. Most damaging of all, the Russians had a chain of agents inside the American atomic weapons development program, and another with access to almost every document of importance which passed between the British and U.S. governments in 1945, including private telegrams sent by Churchill to Presidents Roosevelt and Truman.
Using leads in the decrypted traffic, some of these cases were solved. Maclean was identified as one of the sources of the Churchill telegrams, and many others besides; Klaus Fuchs and the Rosenbergs were unmasked as some of the nuclear spies; while comparison of geographical clues in the decrypts with the movements of Alger Hiss, a senior U.S. State Department official, over a lengthy period made him the best suspect as the agent on Harriman's plane. But despite frenzied counterintelligence and cryptanalytical effort, most of the cryptonyms remain today unidentified.
In Britain the situation was equally grim, but with one major difference. Whereas the Americans had all the Soviet radio traffic passing to and from the USA during and after the war, in Britain Churchill ordered all anti-Soviet intelligence work to cease during the wartime alliance, and GCHQ did not begin taking the traffic again until the very end of the war. Consequently there was far less traffic, and only one break was made into it, for the week September 15 to September 22, 1945, in the Moscow-to-London KGB channel.
There was a series of messages sent to a KGB officer in the London Embassy, Boris Krotov, who specialized in running high-grade agents. The messages came at a time of some crisis for the Russian intelligence services in the West. A young GRU cipher clerk in the Russian Embassy in Canada, Igor Gouzenko, had just defected, taking a mass of material incriminating spies in Canada and the USA, and in Britain a nuclear spy, Alan Nunn May. Most of the messages to Krotov from Moscow Center concerned instructions on how to handle the various agents under his care. Eight cryptonyms were mentioned in all, three of which were referred to as the "valuable ARGENTURA [spy ring] of Stanley, Hicks, and Johnson," two who were routinely referred to together as David and Rosa, and three others. By the end of the week's traffic all contact with the eight spies had been put on ice, and reduced to meetings, except in special circumstances, of once a month.
When I was indoctrinated into VENONA, I remember my first sight of the GCHQ copies of the Moscow-to-London KGB channel. Every time GCHQ broke a few more words in a message, they circulated to the very few users drop copies of the new decrypt. The copies were stamped TOP SECRET UMBRA VENONA, and listed the addresser and addressee, the date and time of the message, the channel and direction (for instance, KGB Moscow/London), and the message priority (whether it was routine or urgent). Underneath would be something like this:
TEXT OF MESSAGE
YOUR COMMUNICATION OF 74689 AND 02985 47199 67789 88005 61971 CONCERNING SPELL H I C K S ENDSPELL 55557 81045 10835 68971 71129 EXTREME CAUTION AT PRESENT TIME 56690 12748 92640 00471 SPELL S T A N L E Y ENDSPELL 37106 72885 MONTHLY UNTIL FURTHER NOTICE. SIGNATURE OF MESSAGE
(This is not a verbatim decrypt; merely a very close approximation to the kind of challenge we were faced with.)
VENONA was the most terrible secret of all, it was incomplete. It was obvious from the decrypts that each of the eight cryptonyms was an important spy, both from the care the Russians were taking to protect them all in September 1945, and because we knew that Krotov specialized in that type of agent. But there was precious little evidence from the traffic which could help us identify them. GCHQ circulated only translations which they had verified, and included the verbatim unsolved groups where they occurred, but they often attached to the copy a separate page of notes giving possible translations of the odd group, which had not yet been verified. Often a message would be repeated several times, as more groups were got out, and it was re-circulated.