Training a Tetrifold Agent with Reinforcement Learning

A small policy. A real question.

Tetrifold is a game we are building around familiar shapes and unfamiliar ways of using them. For this experiment, we focused on Flow Atlas: a mode in which tetrominoes grow a construction that repeatedly folds back into itself. We wanted to know whether a compact learning system could discover useful placement preferences—and how we could tell whether it really improved.

The first trained policy averaged 20.80 folds against 4.85 for a greedy baseline on 20 matched games. A later stage improved a separately measured baseline from 21.23 to 22.80 folds. Another 884,736 training placements then produced no better validated policy. Yet the retained agent could still reach depths 182–188 in longer tests and keep progressing. Each result answers a different question; none should be mistaken for a claim that the game is solved.

Build a square. Fold. Begin again.

Flow Atlas asks the player to complete centered squares: first 3×3, then 5×5, 7×7, and finally 9×9. Completing the largest square triggers a fold. Groups of cells compress into a smaller board, depth advances by one, and the next construction begins from that result. Progress is a sequence of transformations, rather than simply keeping a stack low.

The board can rotate, and locking a piece also produces an automatic turn based on where it lands. Pressure limits how far the construction can extend. A move that fills useful space now can therefore become awkward after rotation. An empty cell sealed inside the board is particularly costly when a future square needs that cell.

Failure rewinds to a checkpoint instead of ending the game. Powers cycle as depth increases: Mend repairs an eligible gap, Reserve changes the available piece, and Precision changes perfect-fold rewards. An agent could keep taking actions and rewinding without getting anywhere. That is why we measure folds, final score, and rewinds within a fixed placement budget, then use longer runs to check for continued progress.

Trust the environment first.

We used Codex to help implement and check a Python port of the repository’s Swift rules. Each action is a placement macro: board turns, piece rotation, a horizontal position, and an optional Reserve choice. The simulator enumerates supported placements and resolves what follows a lock: automatic rotation, square completion, compression, pressure failure, powers, scoring, and any rewind.

The agent reads game state directly. It does not interpret screenshots, learn gestures, or time touch input. A selectable action can still cause a pressure failure; “legal” means that it is available, not that it is safe. Removing animation and input timing lets this experiment concentrate on the geometry of choosing a move.

To check the port, the test suite compiles an oracle from the app’s actual Swift rules and compares its answers with Python. Coverage includes 308 placement/board-turn configurations, 924 randomized spatial-resolution requests, 980 bag draws, and targeted pressure, compression, and Mend cases. The source package records 48 passing tests. That oracle does not execute the full UI store: scoring transactions, Reserve, and rewinds also rely on Python tests and source review, while rendering, persistence, and live input remain outside the learning evaluation.

The decision fits in one line.

Every candidate move produces an afterstate: the board and immediate outcome that would exist after taking it. We describe that result with 25 hand-engineered features. They include target fill, occupied area, radius, perimeter, enclosed holes, narrow pockets, pressure clearance, completed squares, failure, and immediate reward. Most geometric features are normalized by area or extent.

The policy multiplies each feature vector by its learned weights and picks the highest score. We specify what the agent can measure; learning determines how strongly those measurements should matter. It is a linear policy over engineered geometry, with no neural network and no learned visual representation.

The environment exposes the next two pieces, but inference does not search through them. Reserve can make the next visible piece an immediate option when available; that is still a one-move choice, not multi-piece lookahead. NumPy handles scoring. Numba compiles the repeated geometry kernel with @njit(cache=True), and four CPU workers run training episodes. No GPU or neural-network framework is required; we did not run a separate Numba speedup benchmark.

features = extract_features(candidates)
scores = features @ weights
choice = candidates[int(np.argmax(scores))]

Search over policies, not example moves.

Our trainer uses noisy cross-entropy policy search. Szita and Lőrincz applied noisy cross-entropy to Tetris in 2006, introducing noise to counter early convergence. That work provides historical context for the method; our Flow Atlas rules, features, and experiments are not a reproduction of their Tetris study.

The trainer maintains a Gaussian distribution over 25-dimensional weight vectors. In each generation, 24 policies play three shared training seeds. The same seeds make comparisons within a generation less noisy. The six best policies become the elite group. The distribution moves toward their mean and spread, keeping 30% of its previous values and taking 70% from the elites. A standard-deviation floor of 0.15 maintains exploration. Population slots also preserve the current mean and, after initialization, the best saved policy.

The training winner and updated mean are then evaluated on six fixed validation seeds. Only a higher mean validation return replaces the saved best policy. The optimizer separately saves its distribution, random-generator state, and incumbent, so an exact resume can continue the search. A warm start can change the horizon or validation set, but the inherited policy must first be reevaluated under those new conditions.

elite = weights[np.argsort(returns, kind="stable")[-6:]]
mean = 0.3 * mean + 0.7 * elite.mean(axis=0)
std = np.maximum(0.15, 0.3 * std + 0.7 * elite.std(axis=0))

Reward progress, account for setbacks.

The learning signal starts with the actual change in the game’s displayed score, divided by 100. A rewind adds a penalty of 2.0; an ordinary placement costs 0.01. A rewind can therefore both lose score and incur an additional cost. There is no reward for simply remaining active. Episodes end at an external placement limit, reported as truncation rather than a native game over.

This is reinforcement learning through direct policy search: interactions produce episode returns, and those returns determine which weight vectors shape the next generation. There is no supervised dataset of correct moves, no temporal-difference update, and no Q-network. A million simulator placements are interactions across a population, not a million gradient updates to the final policy.

reward = (new_score - old_score) / 100
reward -= 2.0 if rewound else 0.01

First: does learning help?

We froze the initial trained policy and compared it with greedy immediate reward, random legal actions, and untrained zero weights. All four policies played the same 20 seeds, starting at depth zero, with 512 placements per game. The learned policy averaged 20.80 folds, versus 4.85 for greedy. It also rewound much less often: 1.10 times per game against 10.30.

The 4.29× ratio describes mean folds on this sample, not superiority over humans or optimal play. The zero-weight baseline is also not another random player: every action ties, so enumeration order determines its choices. These deliberately simple baselines answer a narrow but useful question—whether learned geometric preferences add value beyond chasing the immediate reward.

Then: can it improve?

Ten additional generations at the original 128-placement training horizon found no validation improvement. We then used 512-placement games, fresh validation seeds, and the inherited search distribution and incumbent. After reevaluating that incumbent at the new horizon, the longer stage found a better checkpoint. Both policies were frozen before a comparison on 30 fresh matched seeds.

Mean folds increased from 21.23 to 22.80. The paired gain was 1.57 folds, with a bootstrap 95% interval of 0.83–2.37. Mean final score rose by about 2,388 points, with an interval of 1,346–3,515. The extended policy completed more folds on 16 seeds, tied on 12, and completed fewer on two. It improved the average without winning every game.

The bootstrap resamples entire matched games, not correlated placements within a game. Its intervals describe seed variation for these fixed policies, not variation across independent training runs; the fold and score intervals are pointwise. The original policy’s earlier 20.80 average and its 21.23 here come from different seeds, not different weights. And the new stage changed more than episode length: extra search, a warm start, validation seeds, and optimizer randomness also changed. This comparison establishes a policy improvement, not its isolated cause.

Useful space can be worth points.

Weights alone are a weak explanation of strategy: the features have different normalizations and can be correlated. Instead, we replayed five matched games per policy and compared their choices on identical states. In those trajectories, narrow target pockets remained after 1,135 of 2,560 placements for the original policy (44.34%), compared with 547 of 2,560 (21.37%) for the extended policy. A pocket here is an empty target cell with at least three occupied cardinal neighbors, distinct from a fully enclosed hole.

One selected state makes the tradeoff tangible. At move six of seed 2,000,100,000’s original-policy trajectory, the original agent places a J for 40 points and leaves one narrow pocket. Given the same starting state, the extended policy chooses 30 points and no pocket, allowing one extra cell outside the target. The lower immediate score preserves a more accessible shape.

A preference, not a guarantee.

At move 144 of seed 2,000,000,000’s original trajectory, the original policy takes ten points while sealing an empty target cell. On the same state, the extended policy takes a zero-point move that leaves no enclosed target hole. This is consistent with valuing future usability over a small immediate score, but does not prove explicit foresight or a universal preference for a larger footprint.

The improved policy still makes the mistake it often avoids. In one replay, move 383 completes a 5×5 square for 340 points while enclosing a cell needed for 7×7. The hole persists through move 407, followed by a rewind at 408. These examples were selected retrospectively, and the five-game behavior study is descriptive. The separate 30-game frozen-policy comparison supports the performance gain; the board examples help interpret it.

More training. The same best policy.

On 13 September, we resumed the strongest optimizer for 24 more generations. That added 884,736 training placements and 147,456 validation placements, taking about 29 minutes with four workers on the recorded machine. No candidate beat the incumbent on validation. The optimizer advanced from generation 16 to 40, but the selected policy remained the checkpoint from generation 15—byte-for-byte identical to the previous best.

A fresh comparison on 30 matched games returned identical episode outcomes for both files: 22.43 mean folds, 33,184.7 points, and 0.33 rewinds. Identical weights on identical seeds should do exactly that. This checks that we retained the same policy; it does not show that every possible continuation would fail. The curve below uses only the coherent longer-stage validation series: the same six reused seeds and a 512-placement horizon.

Count the work once.

Across the four stages, the search ran for 70 generations and 1,751,040 training placements. A further 294,912 validation placements bring the combined total to 2,045,952 environment steps. Recorded training-and-validation invocations sum to about 58 minutes. That timing excludes implementation, tests, separate evaluations, and article preparation, and is not a hardware-independent benchmark.

Continuation logs retain earlier history, so adding every row from every saved log would double-count training. The evidence package checks those shared prefixes and counts only newly executed generations. The resulting plateau belongs to this configuration: hand-engineered observations, a linear model, one-step inference, and a validation protocol with only six reused seeds. None establishes a globally optimal strategy.

What happens after move 512?

The short evaluations had reached only about depth 24. To see whether that was a progression limit, we tested the retained policy on five fresh seeds for 4,096 placements each—eight times the longer training horizon. Final depths were 186, 182, 188, 187, and 188. Every run was still folding at the cap, with 12–13 folds in its final 256 placements.

We defined a stall in advance as 128 consecutive placements without a fold and continued through setbacks to observe recovery. No run reached that threshold. The longest delay was 62 placements at depth 17; the next fold came at placement 431. First rewinds occurred at depths ranging from 17 to 163, and the agent recovered from them. The checkpoint hash confirms that these runs tested the exact policy retained after the final continuation.

Pressure stops tightening at depth 18 in standard Flow. Powers continue cycling, but the rules define no terminal depth, so the simulator always reports terminated=False. That flag is not evidence of survival. Continued fold events demonstrate progress. Five finite runs cannot prove indefinite play or exclude a rarer stall; they show that the earlier observed depth was not where this agent became unable to continue.

A baseline we can build on.

The implementation lives in Tetrifold’s rl/ directory and uses Python 3.10 or later, NumPy, and Numba, with pytest for checks. A saved policy can export deterministic replays. Resuming training needs the optimizer state as well as the best weights. The accompanying local research package freezes the experiment artifacts and provides figure sources, seed definitions, and behavior-analysis scripts. The test counts in this article describe that recorded experiment, not a new training run performed for publication.

For us, the useful outcome is a working, measured baseline. Twenty-five learned weights can encode geometric preferences that outperform immediate-reward greed, but they do so inside a carefully constructed environment with a narrow observation and action interface. The research is useful because its assumptions are visible. A next experiment could test lookahead, richer features, or a broader validation protocol—but it should change a stated hypothesis and measure the result, rather than assume a larger model must be better.