Editing CommunityData:TACC (section)

=== Install <code>vllm</code> into your virtual environment ===

# Get an H100 node by running <code>idev -p h100 -t 48:00:00</code>
# Run <code> module load gcc/13.2.0; module load cuda </code> to load the nvidia module which puts the Nvidia cuda compilers (<code>nvc</code> and <code>nvcc</code>) on your <code>$PATH</code>.  
# Navigate to the the directory where you created your virtual environment and install vllm by running <code>uv pip install vllm[flashinfer] --torch-backend=auto --no-cache</code> and then.
<!--- # Run <code> module load gcc/14.2.0 </code> to load a modern version of the C compiler <code>gcc</code>. --->
# Test your installation by seeing if you can run the <code>gpt-oss-20B</code> model in vllm. <code>uv run vllm serve openai/gpt-oss-20b  --async-scheduling</code>
<!---
==== Step 3 Explanation ====
The vllm installation statically links the Nvidia cuda compilers (<code>nvc</code> and <code>nvcc</code>). This means that you don't need these on your <code>$PATH</code> after the installation. However, the default <code>gcc</code> on Stampede 3 is out of date. It doesn't support features that vllm uses when it tries to compile cuda graphs, causing an error. By following the above steps the cuda compilers will be installed with <code>vllm</code> but running with a version of gcc that works. These issues seemed like it might have been related to conflicts between the container and TACC environmen, but actually it is not.>

=== Load GCC/14.2.0 by default ===
You might prefer to load gcc 14 only when you need it. But if that gets annoying you can load it automatically. Just add
<code>module load gcc/14.2.0</code> to your [https://unix.stackexchange.com/questions/129143/what-is-the-purpose-of-bashrc-and-how-does-it-work .bashrc].
--->