puffer eval/train #487
Description
Traceback (most recent call last):
File "/opt/puffer/pufferlib/pufferl.py", line 37, in <module>
from pufferlib import _C
ImportError: /opt/puffer/pufferlib/_C.cpython-313-x86_64-linux-gnu.so: undefined symbol: static_vec_log
pip show torch triton #ubuntu 25.10
Name: torch
Version: 2.6.0+cu124
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3-Clause
Location: /home/garner/.local/lib/python3.13/site-packages
Requires: filelock, fsspec, jinja2, networkx, nvidia-cublas-cu12, nvidia-cuda-cupti-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-runtime-cu12, nvidia-cudnn-cu12, nvidia-cufft-cu12, nvidia-curand-cu12, nvidia-cusolver-cu12, nvidia-cusparse-cu12, nvidia-cusparselt-cu12, nvidia-nccl-cu12, nvidia-nvjitlink-cu12, nvidia-nvtx-cu12, setuptools, sympy, triton, typing-extensions
Required-by: heavyball, linear-operator, pufferlib
---
Name: triton
Version: 3.2.0
...
04cb2f1 merges 4202c66 (or rebase with 88ee47f as HEAD)
python setup.py build_ext --inplace --force #ubuntu 25.10
running build_ext
running build_torch
/home/garner/.local/lib/python3.13/site-packages/torch/utils/cpp_extension.py:458: UserWarning: There are no x86_64-linux-gnu-g++ version bounds defined for CUDA version 12.4
warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
building 'pufferlib._C' extension
/home/garner/.local/lib/python3.13/site-packages/torch/utils/cpp_extension.py:2059: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
...
No error - 5cbaf88 - as 3.0's puffer eval, but 3.0's puffer train:
BackendCompilerFailed: backend='inductor' raised:
ImportError: cannot import name 'Language' from 'triton.backends.compiler'
(/home/garner/.local/lib/python3.13/site-packages/triton/backends/compiler.py)
Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
You can suppress this exception and fall back to eager by setting:
import torch._dynamo
torch._dynamo.config.suppress_errors = True
Found CUDA Toolkit Archive, if needed.
sudo apt-get install python3-numpy && pip install rich-argparse==1.7.1 as ubuntu 24.04:
Sat Feb 28 16:39:25 2026
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.288.01 Driver Version: 535.288.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce GTX 980 Off | 00000000:01:00.0 Off | N/A |
| 0% 56C P8 23W / 216W | 1MiB / 4096MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
sudo apt-get install nvidia-cuda-toolkit && python setup.py build_ext --inplace --force && pip install -e . --no-build-isolation --break-system-packages && puffer train puffer_target on torch 2.6.0+cu124:
...
BackendCompilerFailed: backend='inductor' raised:
RuntimeError: Found NVIDIA GeForce GTX 980 which is too old to be supported by the triton GPU compiler, which is used as the
backend. Triton only supports devices of CUDA Capability >= 7.0, but your device is of CUDA capability 5.2
Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
You can suppress this exception and fall back to eager by setting:
import torch._dynamo
torch._dynamo.config.suppress_errors = True
puffer eval puffer_target --load-model-path=latest
Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.
Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.
See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.
INFO: Initializing raylib 5.5
INFO: Platform backend: DESKTOP (GLFW)
INFO: Supported raylib modules:
INFO: > rcore:..... loaded (mandatory)
INFO: > rlgl:...... loaded (mandatory)
INFO: > rshapes:... loaded (optional)
INFO: > rtextures:. loaded (optional)
INFO: > rtext:..... loaded (optional)
INFO: > rmodels:... loaded (optional)
INFO: > raudio:.... loaded (optional)
INFO: DISPLAY: Device initialized successfully
INFO: > Display size: 2560 x 1440
INFO: > Screen size: 1080 x 720
INFO: > Render size: 1080 x 720
INFO: > Viewport offsets: 0, 0
INFO: GLAD: OpenGL extensions loaded successfully
INFO: GL: Supported extensions count: 384
INFO: GL: OpenGL device information:
INFO: > Vendor: NVIDIA Corporation
INFO: > Renderer: NVIDIA GeForce GTX 980/PCIe/SSE2
INFO: > Version: 3.3.0 NVIDIA 535.288.01
INFO: > GLSL: 3.30 NVIDIA via Cg compiler
INFO: GL: VAO extension detected, VAO functions loaded successfully
INFO: GL: NPOT textures extension detected, full NPOT textures supported
INFO: GL: DXT compressed textures supported
INFO: GL: ETC2/EAC compressed textures supported
INFO: PLATFORM: DESKTOP (GLFW - X11): Initialized successfully
INFO: TEXTURE: [ID 1] Texture loaded successfully (1x1 | R8G8B8A8 | 1 mipmaps)
INFO: TEXTURE: [ID 1] Default texture loaded successfully
INFO: SHADER: [ID 1] Vertex shader compiled successfully
INFO: SHADER: [ID 2] Fragment shader compiled successfully
INFO: SHADER: [ID 3] Program shader loaded successfully
INFO: SHADER: [ID 3] Default shader loaded successfully
INFO: RLGL: Render batch vertex buffers loaded successfully in RAM (CPU)
INFO: RLGL: Render batch vertex buffers loaded successfully in VRAM (GPU)
INFO: RLGL: Default OpenGL state initialized successfully
INFO: TEXTURE: [ID 2] Texture loaded successfully (128x128 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Default font loaded successfully (224 glyphs)
INFO: SYSTEM: Working Directory: /media/garner/puffer
INFO: TIMER: Target time per frame: 16.667 milliseconds
INFO: FILEIO: [resources/shared/puffers_128.png] File loaded successfully
INFO: IMAGE: Data loaded successfully (256x128 | R8G8B8A8 | 1 mipmaps)
INFO: TEXTURE: [ID 3] Texture loaded successfully (256x128 | R8G8B8A8 | 1 mipmaps)
INFO: FILEIO: [resources/target/star.png] File loaded successfully
INFO: IMAGE: Data loaded successfully (128x128 | R8G8B8A8 | 1 mipmaps)
INFO: TEXTURE: [ID 4] Texture loaded successfully (128x128 | R8G8B8A8 | 1 mipmaps)
git checkout 2034c81 --force #4.0-dev
python setup.py build_ext --inplace --force && pip install -e . --no-build-isolation --break-system-packages && puffer train puffer_target
Traceback (most recent call last):
File "/media/garner/puffer/setup.py", line 14, in <module>
import pybind11
ModuleNotFoundError: No module named 'pybind11'
garner@os:/opt/puffer$ sudo apt-get install python3-pybind11
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
libeigen3-dev pybind11-dev
Suggested packages:
libeigen3-doc libmpfrc++-dev pybind11-doc
The following NEW packages will be installed:
libeigen3-dev pybind11-dev python3-pybind11
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 1382 kB of archives.
After this operation, 10.9 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://us.archive.ubuntu.com/ubuntu noble/universe amd64 pybind11-dev all 2.11.1-2 [159 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu noble/universe amd64 python3-pybind11 all 2.11.1-2 [167 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu noble-updates/universe amd64 libeigen3-dev all 3.4.0-4build0.1 [1057 kB]
Fetched 1382 kB in 1s (1579 kB/s)
Selecting previously unselected package pybind11-dev.
(Reading database ... 169132 files and directories currently installed.)
Preparing to unpack .../pybind11-dev_2.11.1-2_all.deb ...
Unpacking pybind11-dev (2.11.1-2) ...
Selecting previously unselected package python3-pybind11.
Preparing to unpack .../python3-pybind11_2.11.1-2_all.deb ...
Unpacking python3-pybind11 (2.11.1-2) ...
Selecting previously unselected package libeigen3-dev.
Preparing to unpack .../libeigen3-dev_3.4.0-4build0.1_all.deb ...
Unpacking libeigen3-dev (3.4.0-4build0.1) ...
Setting up pybind11-dev (2.11.1-2) ...
Setting up libeigen3-dev (3.4.0-4build0.1) ...
Setting up python3-pybind11 (2.11.1-2) ...
Processing triggers for man-db (2.12.0-4build2) ...
garner@os:/opt/puffer$ python setup.py build_ext --inplace --force && pip install -e . --no-build-isolation --break-system-packages && puffer train puffer_target
configuration error: `project.license` must be valid exactly by one definition (2 matches found):
- keys:
'file': {type: string}
required: ['file']
- keys:
'text': {type: string}
required: ['text']
DESCRIPTION:
`Project license <https://peps.python.org/pep-0621/#license>`_.
GIVEN VALUE:
"MIT"
OFFENDING RULE: 'oneOf'
DEFINITION:
{
"oneOf": [
{
"properties": {
"file": {
"type": "string",
"$$description": [
"Relative path to the file (UTF-8) which contains the license for the",
"project."
]
}
},
"required": [
"file"
]
},
{
"properties": {
"text": {
"type": "string",
"$$description": [
"The license of the project whose meaning is that of the",
"`License field from the core metadata",
"<https://packaging.python.org/specifications/core-metadata/#license>`_."
]
}
},
"required": [
"text"
]
}
]
}
Traceback (most recent call last):
File "/media/garner/puffer/setup.py", line 450, in <module>
setup(
File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 107, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 159, in setup
dist.parse_config_files()
File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 899, in parse_config_files
pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
File "/usr/lib/python3/dist-packages/setuptools/config/pyprojecttoml.py", line 66, in apply_configuration
config = read_configuration(filepath, True, ignore_option_errors, dist)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/setuptools/config/pyprojecttoml.py", line 127, in read_configuration
validate(subset, filepath)
File "/usr/lib/python3/dist-packages/setuptools/config/pyprojecttoml.py", line 55, in validate
raise ValueError(f"{error}\n{summary}") from None
ValueError: invalid pyproject.toml config: `project.license`.
configuration error: `project.license` must be valid exactly by one definition (2 matches found):
- keys:
'file': {type: string}
required: ['file']
- keys:
'text': {type: string}
required: ['text']
sudo apt-get install ninja-build and added to setup.py: os.environ['TORCH_CUDA_ARCH_LIST'] = "5.2".
To close, may need changes as static_vec_log ImportError remains. Triton ended maxwell support before pufferlib 1.0, meaning pufferlib hasn't always used triton. Assuming 4.0 will no longer rely upon torch and therefore triton, perhaps pufferlib itself may need a compute capability check as an incompatibility may manifest in the form of a nvcc linker error, right? Eg. since ImportError was at runtime and maxwell/pascal support may still be possible.