HEVC NVENC with ffmpeg: '-pix_fmt p010le' and '-pix_fmt yuv420p10le' doesn't work

HEVC NVENC in FFmpeg requires the input format to match the encoder's native pixel format, typically yuv420p for 8-bit or yuv420p10le for 10-bit. The error occurs because p010le is not a valid output pixel format for the hevc_nvenc encoder, which expects planar YUV data rather than packed formats. You must ensure your input stream is converted to the correct format using the -pix_fmt option before encoding.

Understanding Pixel Format Compatibility

The core issue when using hevc_nvenc is a mismatch between the source video's pixel format and what NVIDIA’s hardware encoder accepts. NVIDIA encoders are highly optimized but strict about input formats. For HEVC (H.265) encoding, the standard 8-bit format is yuv420p, while 10-bit high dynamic range (HDR) content requires yuv420p10le. When users attempt to use -pix_fmt p010le, they are specifying a packed 10-bit format. The hevc_nvenc encoder does not support packed pixel formats as direct inputs. It requires planar formats where the luma (Y) and chroma (U/V) components are stored separately. This is why yuv420p10le works, but p010le fails with an "Invalid argument" or "Unsupported pixel format" error. It is crucial to distinguish between the pixel format of the input file and the pixel format specified for the encoder. If your source file is already in p010le, you cannot simply pass it to the encoder. You must insert a software conversion step to transform the data into yuv420p10le before handing it off to the GPU encoder.

Correcting the Command Line Syntax

To resolve this, you need to explicitly set the pixel format that matches your desired bit depth. For standard SDR (Standard Dynamic Range) video, use -pix_fmt yuv420p. For HDR or high-quality archival, use -pix_fmt yuv420p10le. Here is the corrected structure for a basic HEVC NVENC command:

ffmpeg -i input.mp4 -c:v hevc_nvenc -pix_fmt yuv420p10le -b:v 10M output.mp4

If your input file uses a different format, such as rgb24 or p010le, FFmpeg will automatically convert it if you specify the target -pix_fmt. However, if you are piping data or using complex filters, you may need to be more explicit. For example, if you are dealing with a 10-bit source that is incorrectly formatted, you can force the conversion:

ffmpeg -i input_10bit.p010le -vf format=yuv420p10le -c:v hevc_nvenc -pix_fmt yuv420p10le output.mp4

In this command, the -vf format=yuv420p10le ensures that the video filter chain outputs the correct planar format before the encoder processes it. This avoids the common pitfall of assuming the encoder will handle arbitrary input formats.

Handling 8-Bit vs 10-Bit Content

Choosing between 8-bit and 10-bit encoding depends on your source material. Most web-based video and standard broadcast content is 8-bit. Using yuv420p10le for 8-bit content increases file size without visible quality improvement. Conversely, encoding 10-bit HDR content as 8-bit results in banding and loss of detail in shadows and highlights. Check your input file properties using:

ffprobe -v error -select_streams v:0 -show_entries stream=width,height,r_frame_rate,pix_fmt -of default=noprint_wrappers=1:nokey=1 input.mp4

Look at the pix_fmt field. If it says yuv420p, stick to 8-bit encoding. If it says yuv420p10le or p010le, you should use 10-bit encoding. Note that some older NVIDIA drivers or cards may have limited support for 10-bit NVENC, so verify your GPU capabilities. For more details on optimizing HEVC settings, see FFMPEG: H.265 (HEVC) Video Encoding with VCNVENC (yuv420) at 800Mbps.

Diagnostics and Troubleshooting

If you still encounter errors after adjusting the pixel format, check the following:

• Driver Version: Ensure your NVIDIA drivers are up to date. Older drivers may lack support for specific HEVC profiles or 10-bit processing.
• GPU Memory: 10-bit encoding consumes more VRAM. If you run out of memory, the encode will fail. Reduce resolution or bitrate if necessary.
• Profile Selection: Use -profile:v main10 for 10-bit content. Standard profiles like main are for 8-bit and may reject 10-bit inputs.

A complete, robust command for 10-bit HEVC NVENC looks like this:

ffmpeg -i input.mp4 -c:v hevc_nvenc -profile:v main10 -pix_fmt yuv420p10le -rc vbr -b:v 10M output.mp4

This sets the profile to main10, specifies the 10-bit pixel format, and uses variable bitrate encoding for better quality efficiency. For general guidance on obtaining specific FFmpeg settings, refer to Attempting to Get Specific Settings for FFmpeg Encodes.

Frequently Asked Questions

Why does my encode fail with 'Invalid pixel format'?

This error usually means the input format does not match the encoder's expectations. Ensure you are using yuv420p for 8-bit or yuv420p10le for 10-bit, and avoid packed formats like p010le.

Can I use NVENC for 10-bit video?

Yes, most modern NVIDIA GPUs support 10-bit HEVC encoding via NVENC. You must specify -pix_fmt yuv420p10le and use the main10 profile to enable this functionality.

What is the difference between yuv420p and yuv420p10le?

yuv420p is an 8-bit planar format suitable for standard dynamic range video. yuv420p10le is a 10-bit planar format used for HDR content, providing greater color depth and reducing banding artifacts.

How do I check my current pixel format?

Use the ffprobe command with the -show_entries stream=pix_fmt flag to inspect the pixel format of your source media file before encoding.

Does NVENC support other pixel formats?

NVENC primarily supports yuv420p, yuv420p10le, and sometimes nv12 depending on the codec and version. Packed formats like p010le are generally not supported directly by the hardware encoder.