[Bug Report] torch 2.8.0 makes model generation go haywire

[hijohnnylin]

Describe the bug

• model.generate works fine on torch 2.7.1 but in 2.8.0 (on macos at least), it goes haywire

• workaround is to force version <=2.8.0 on macos

• macos 15.6.1

• python 3.11.8

I'll dig into this more later (eg test if it affects linux too) but just flagging this for now so it doesn't get lost.

System Info

• I have checked that there is no similar issue in the repo (required)

[gaabrielfranco]

I had the same problem. I was going crazy trying to find it. Forcing torch back to 2.7.1 also worked for me.

[hijohnnylin]

@gaabrielfranco were you on cuda or mps or cpu?

[gaabrielfranco]

@hijohnnylin I was using mps.

[noahrossi]

I ran into this exact problem as well and it seems to be an upstream issue w/ pytorch: pytorch/issues/161640 . It's fixed in pytorch 2.9.0, and the issue doesn't appear (at least for my reproduction) in 2.7 or 2.9.