ChatGPT-4o's stock picks contradict each other in 1,359 three-firm loops, a working paper finds

Counting longer loops, they report 13,155,050 cycles covering three to seven firms. Because cycles overlap and share the same pairwise answers, the authors note that millions of cycles could come from a small number of inconsistent answers. Even so, at least 181 of the 4,950 recommendations (3.66%) would have to change before the answers fit a single ranking.

The authors say this matters for anyone who delegates a series of pairwise trades to an AI adviser. A broker could offer a trade sequence that returns the investor to the starting holding minus a fee at each step. They also argue that fixing the cycles after the fact has no principled basis and is computationally hard.

The paper's theory holds that a Transformer trained by maximum likelihood produces output probabilities that converge to the empirical vote shares of its training text as model capacity grows. The authors say the same logic extends to prompts outside the training data by way of similarity in the model's meaning space. The investment test used ChatGPT-4o, whose design and training data are proprietary.

To check the vote-share claim, the authors built small two-layer Transformers with embedding dimension 5 and trained each on 1,000 examples of the question "Which stock is better: Tesla or Nvidia?". Across eleven training sets, with the Nvidia answer share ranging from 0% to 100%, the learned probabilities matched the share closely. At a 30% share the model gave 0.299, and at 50% it gave 0.496.

On fixing the problem, the authors argue that cycles cannot be removed without arbitrarily favoring some groups of authors in the training data. They add that asking about three or more stocks at once does not help, since users still ask pairwise questions. Finding the closest consistent ranking is NP-hard, so they say correction cannot be a routine post-processing step.

On temperature, they show that it acts as a power-law transformation of vote shares. Using U.S. Census data as an illustration, they calculate that a 57% group receives 72% of output probability at temperature 0.7 while smaller groups are pushed down. They also compare ChatGPT-4o's outputs at varied temperatures with the probabilities their theory implies.