Information Theory in Pokémon: The Time Capsule

Special thanks to my brother for introducing me to this topic, and Bulbapedia for providing the technical details needed to write this post.

The first two Pokémon games were released simultaneously in the US in 1998. One of their key features was the ability to connect with other Pokémon players and either battle or trade Pokémon. (At the time, this was a much more revolutionary concept than it may seem today.)

Just two years later, Nintendo released two sequel games, called Pokémon Gold and Silver, which featured new areas, creatures, attacks, and game mechanics. Gold and Silver were also trade-compatible with Red and Blue, meaning players could trade between the older games and the newer ones.

Yet Gold and Silver had features that Red and Blue lacked. So how did Nintendo and Game Freak maintain both backwards and forwards compatibility between games that could not be patched? And how did their solution affect the way Pokémon is played?

Note: from here on, I will be referring to Red and Blue as "Gen 1," and Gold and Silver as "Gen 2."1

It should also be noted that not all Pokémon can be traded between games. For example, there is no data in Gen 1 games for the new Pokémon intorduced in Gen 2. So these Pokémon cannot be traded using the Time Capsule. This rule is enforced by the Gen 2 game, which includes a subroutine that checks to determine whether your Pokémon will be compatible or not. (You can exploit a glitch to trade anything, but it results in strange behavior on the Gen 1 receiver.)

An easy solution would be something like this:

This solution, although backwards compatible (Pokémon from Gen 2 are converted to Gen 1 successfully), is not forwards compatible (Pokémon from Gen 1 are not converted back to Gen 2 successfully, because the random number generator is not guaranteed to have the same result). With this method, trading a Pokémon from Gen 1 to Gen 2 then back again does not necessarily give you the same Pokémon you started with.

So Gen 2 Pokémon must be able to store all their information in the Gen 1 data structure, which does not allocate space for this information.

To understand how they do this, let's take a look at the structures used by each generation.

Gen 1 Datastructure

Offset Size Description
0x00 1 byte Species identifier
0x01 2 bytes Current HP
0x03 1 byte Level
0x04 1 byte Status condition
0x05 2 bytes Type (1 byte each)
0x07 1 byte Catch rate
0x08 4 bytes Moves (1 byte each)
0x0C 2 bytes Original trainer ID
0x0E 3 bytes Experience
0x11 10 bytes EV data (HP, Attack, Defense, Speed, Special @ 2 bytes each)
0x18 2 bytes IV data
0x1D 4 bytes Move PP (1 byte each)
0x21 1 byte Level
0x22 10 bytes Stat levels (HP, Attack, Defense, Speed, Special @ 2 bytes each)

44 bytes total

Gen 2 Data Structure

Offset Size Description
0x01 1 byte Species
0x01 1 byte Held item
0x02 4 bytes Moves (1 byte each)
0x06 2 bytes Original trainer ID
0x08 3 bytes Experience
0x0B 10 bytes EV data (HP, Attack, Defense, Speed, Special @ 2 bytes each)
0x15 2 bytes IV data
0x17 4 bytes Move PP (1 byte each)
0x1B 1 byte Friendship value
0x1C 1 byte Pokerus
0x1D 2 bytes Caught data
0x1F 1 byte Level
0x20 1 byte Status condition
0x21 1 byte Unused
0x22 2 bytes Current HP
0x24 12 bytes Stats (HP, Attack, Defense, Speed, Special Attack, Special Defense) @ 2 bytes each)

48 bytes total

Gen 2 does not store 3 properties from the Gen 1 table: catch rate (1 byte), type (2 bytes), and the redundant level entry (1 byte). On top of that, Gen 2 has 7 bytes of data with no corresponding position in the Gen 1 data structure: held item (1 byte), friendship (1 byte), Pokerus (1 byte), caught data (2 bytes), and splitting the Special stat into Special Attack and Special Defense (2 extra bytes). (If you include the unused byte, the data structure for Gen 2 has 8 bytes worth of information not used in Gen 1).

So, to transfer a Pokémon from Gen 2 to Gen 1, Gen 2 has to store 8 bytes of information in 4 bytes. How do they do this?

Note: 4 bytes is generous. Keep in mind the data structure still needs to be useable by Gen 1 games, so we can't corrupt any of it with "incorrect" data.

Step 1: Be smart about what you store

One easy way to tackle this problem is to reduce the amount of information you need to store in the first place. We can't pack 8 bytes into 4, but we might find a way to pack 6.

In the transfer from Gen 2 to Gen 1, friendship, Pokerus, and caught data is lost. This saves a total of 4 bytes.

Upon trading back from Gen 1 to Gen 2, friendship is reset to its default value (0x46), Pokerus is turned off (0x00), and caught data is reset (0x0000).

It's worth noting that resetting this data makes some level of intuitive sense. Your Pokémon has just spent some time with someone else, so you might have to work your way back to becoming friends again. One could argue that something about the Time Capsule cures Pokerus. Upon trading back, the caught data is reset to say your Pokémon came from a Gen 1 trade, which is true.

Excluding the unused byte, we now only have to store held item (1 byte) and deal with splitting Special into Special Attack and Special Defense (2 bytes each). Each of these problems has its own solution.

Step 2: Exploit superfluous data

Sometime between Gen 1 and Gen 2, the Pokémon developers realized they did not need to store Catch Rate on already-caught Pokémon. Catch Rate determines how difficult it is to catch a Pokémon, which can only be done once per Pokémon. Once a Pokémon is caught, the data becomes useless. Every species of Pokémon has its own catch rate - not each individual Pokémon.

In Gen 2, Catch Rate is stored on a per-species basis within the game itself, rather than being part of the data stored on each individual Pokémon. This frees up space in the Gen 2 data structure, and can safely be considered unused in Gen 1, since Catch Rate data is never accessed on a caught Pokémon.

Catch Rate is only 1 byte in the Gen 1 data structure, which makes it a prime candidate for storing a Pokémon's held item. Since Gen 1 doesn't support held items, you cannot interact with your transferred Pokémon's held item anyway. But it'll still be there when you trade it back to Gen 2.

Step 3: Smart formula design

All that's left to deal with at this point is the split of Special into Special Attack and Special Defense. This seems like it requires 2 extra bytes, but it can actually be done in 0 bytes, since the formula for Pokémon stats are deterministic. Stat data is generated using EV data, IV data, and Base Stat data particular to each species of Pokémon.

Special Attack and Special Defense share a single Special EV, and all stats share the same IV, unique to the individual Pokémon. Special Attack and Special Defense are only differentiated in Gen 2 by the species' Base Stat data. This allows us to generate both stats from one stored EV.

As a result, Special Attack and Special Defense don't need to be included at all in the transfer between Gen 1 and Gen 2, since each game can generate its version of stats deterministically. All Gen 1-compatible Pokémon in Gen 2 have the same Base Stats for HP, Attack, Defense, and Speed that they did in Gen 1, so no stats change in the transfer, with the exception of splitting and combining Special Attack and Special Defense.

Consequences

What interested me about this topic was not so much their clever solutions to this problem, but the consequences that result from those decisions.

Since catch rate and held item share a slot in the data structure, all Gen 1 Pokémon are given a held item when traded through the Time Capsule, even though Gen 1 has no support for held items. Which held item a Pokémon gets will be determined by its Catch Rate, which is determined by its species. So, for example, every Pikachu will have the same held item when traded from Gen 1 to Gen 2.

In order to accommodate the Time Capsule, Gen 2 developers had to be very selective about what data they stored with each Pokémon. Note that some properties of Gen 2 Pokémon, such as shininess and gender, are not stored in the data structure. I speculate that this was done because they knew they would be unable to maintain this data across the Time Capsule. Instead, this data is determined by each Pokémon's IV data, which is randomly generated when the Pokémon is first encountered.

IV data is 2 bytes, and stores 4 stats: Attack, Defense, Special, and Speed. Each stat is stored within one nibble, and ranges from 0-15. A Pokémon's HP IV is calculated from the other four IVs.

A Pokémon is shiny when its Defense, Speed, and Special IVs have value 10 (0xA), and its Attack IV is 2, 3, 6, 7, 10, 11, 14, or 15. There is a 1/8192 chance (0.01221%) that a Pokémon will be shiny.

A Pokémon's gender is determined by its Attack IV. Each species has one of four gender ratios, represented by a single number between 0 and 15. If a Pokémon's Attack IV is less than or equal to its gender value, it is female; otherwise, it is male. This has a few consequences.

First, by having the formula work this way, female Pokémon always have a lower Attack stat than male Pokémon of the same species. Within a species, the maximum Attack stat for a female Pokémon will always be lower than the minimum Attack stat for a male Pokémon. Furthermore, in high-level Pokémon battling, everyone uses Pokémon that are engineered to be as perfect as possible. This means female Pokémon are never used in serious competitive play if their species has a male counterpart. This is... well, pretty sexist! In Gen 2, female Pokémon are hard-coded to be inferior to male Pokémon because the developers did not carefully consider the consequences of using IV data to determine gender. (Or they considered it but didn't care.)

Second, species that are unlikely to be female (12.5% chance, gender value 0x1) can never have a shiny female, since the lowest Attack IV that results in a shiny Pokémon is 2, and 0x2 > 0x1.

Thankfully, both of these gender consequences were remedied in Gen 3, where the developers introduced a new value to the Pokémon data structure called Personality Value. This is used to determine virtually all non-stat differences between Pokémon, and decouples stats from gender.

But Gen 3 is not compatible with Gen 2 or Gen 1, partially for hardware reasons.