We provided several resources related to parallel programming. The original post of CUDA programming resources was published in July 2009. Since then, the CUDA programming has gained a lot of steam and proven that a wide variety of computing applications can benefit from GPGPUs. Before going to CUDA resources, here are the various programming tutorial resources we published so far.
Basic parallel computing tutorial
MapReduce tutorial
Cell processor programming
OpenMP tutorial
PThreads Tutorials
Intel Threading Building Blocks
MPI
CUDA (Original post, updated here)
OpenCL

In this article, we look at Compute Unified Device Architecture (CUDA) from NVIDIA to program inexpensive multi-threaded GPUs. CUDA includes the CUDA Instruction Set Architecture (ISA) and the parallel compute engine in the GPU. To program to the CUDA architecture, developers can, today, use C, one of the most widely used high-level programming languages, which can then be run at great performance on a CUDA enabled processor. Other languages will be supported in the future, including FORTRAN and C++.

The best resource for CUDA related tutorials and introduction is Nvidia’s CUDA Zone itself.

Nvidia’s CUDA Training page is here.

Professors Wen-mei Hwu and David Kirk taught ECE 498AL at University of Illinois @ Urbana-Champaign. Here are the course materials (including slides and voice).

Here are some lectures and presentations that we posted on MulticoreInfo.
Advanced algorithmic techniques for GPUs (Video Lectures) (by Prof. Wen-mei Hwu)
NVIDIA presentation with the new features of CUDA 4.0
Slides from Manycore and Accelerator-based HPC Tutorials Online
Optimizing Data Movement in the PGI Accelerator Programming Model
Strategies for Parallelism by UIUC’s Professor Wen-mei Hwu

Other resources
GPU Computing Online Seminars

Dr. Dobb’s CUDA, Supercomputing for the Masses series of articles by Rob Farber, a senior scientist at Pacific Northwest National Laboratory, is a good source for introducing yourself to CUDA.

• CUDA, Supercomputing for the Masses: Part 1
• CUDA, Supercomputing for the Masses: Part 2
• CUDA, Supercomputing for the Masses: Part 3
• CUDA, Supercomputing for the Masses: Part 4
• CUDA, Supercomputing for the Masses: Part 5
• CUDA, Supercomputing for the Masses: Part 6
• CUDA, Supercomputing for the Masses: Part 7
• CUDA, Supercomputing for the Masses: Part 8
• CUDA, Supercomputing for the Masses: Part 9
• CUDA, Supercomputing for the Masses: Part 10
• CUDA, Supercomputing for the Masses: Part 11
• CUDA, Supercomputing for the Masses: Part 12
• CUDA, Supercomputing for the Masses: Part 13
• CUDA, Supercomputing for the Masses: Part 14
• CUDA, Supercomputing for the Masses: Part 15
• CUDA, Supercomputing for the Masses: Part 16
• CUDA, Supercomputing for the Masses: Part 17
• CUDA, Supercomputing for the Masses: Part 18
• CUDA, Supercomputing for the Masses: Part 19
• CUDA, Supercomputing for the Masses: Part 20

CUDA, Supercomputing for the Masses: Part 21

CUDA Tutorials from Supercomputing Blog
Search algorithm with CUDA from Supercomputing Blog

CUDA Programming Talk at Winter School in Parallel Computing by Johan Seland

Nvidia Tutorial CUDA by Cyril Zeller

Stanford University course on Programming Massively Parallel Processors with CUDA

GPU Technology Conference (GTC) 2010 Presentation Archives
GPU Technology Conference (GTC) 2009 Presentation Archives

CUDA SDK Code Samples: The CUDA Developer SDK provides examples with source code, utilities, and white papers to help you get started writing software with CUDA. The SDK includes dozens of code samples covering a wide range of applications.

CUDA C Programming Guide, version 3.2
CUDA C Best Practices Guide, version 3.2
CUDA Reference Manual, version 3.2
CUDA-GDB: CUDA Debugger User Manual
Tuning CUDA Applications for Fermi

CUDA Libraries
CULA: GPU-accelerated LAPACK libraries
PGI CUDA Fortran Compiler
jcuda: Java bindings for CUDA
cudpp: CUDA Data Parallel Primitives Library
FORTRAN bindings for CUDA from Hoopoe
GPULib from Tech-X: A library of mathematical functions that facilitate the use of HPC resources
CUDALink: Allows Mathematica to use the CUDA architecture on GPUs
Jacket for Matlab from Accelereyes
libJacket for C/C++ from Accelereyes
CUDA.NET from hoopoe
CUDA with Perl, Ruby, Python: KappaCUDA
CUDA parallel computation API from Python: PyCUDA
CUBLAS Library: GPU-accelerated BLAS library
CUFFT Library: GPU-accelerated FFT library
CUSPARSE Library: GPU-accelerated Sparse Matrix library
CURAND Library: GPU-accelerated RNG library

CUDA Books

Nvidia’s CUDA University have links to Training Courses, Slides from University of Illinois’ ECE 498AL course taught by Professor Wen-mei W. Hwu and David Kirk, NVIDIA Chief Scientist.
Nvidia CUDA U

NVidia presented a great tutorial on CUDA programming at the HPCMP User’s Group conference in June 2009, that has 192 slides. You can view those slides here.

CUDA Tutorial presented at ISC 2009:

• Introduction to GPU Computing (PDF)
• Basic CUDA (PDF)
• CUDA Toolkit & Libraries (PDF)
• CUDA Optimization (PDF)
• Introduction to OpenCL (PDF)
• Case Study: Computational Fluid Dynamics (PDF)

Source: GPGPU.org

Another interesting and simple tutorial: CUDA Tutorial at Parallel Panorama
CUDA Programming and Development Forum at NVIDIA CUDA Zone
The PGI Accelerator Programming Model on NVIDIA GPUs by Michael Wolfe, PGI Compiler Engineer