Understanding the Aspect Ratio Challenge
When combining vertical (portrait) and horizontal (landscape) video content into a single image, the primary technical hurdle is the mismatch in dimensions. Portrait videos typically have a 9:16 aspect ratio, while landscape videos follow a 16:9 ratio. If you attempt to stack these directly using the hstack filter without preprocessing, FFmpeg will fail because the filter requires all input streams to have identical heights. A naive approach might scale both to an arbitrary small number, such as 180 pixels, but this ignores the fact that the resulting widths will differ significantly, potentially leading to distorted visuals or misaligned elements if not handled correctly.
To solve this, we must normalize the height first. By forcing both inputs to share the same pixel height, we guarantee compatibility with the stacking operation. The width will naturally vary based on the aspect ratio, which is acceptable since hstack only cares about vertical alignment. We can then add padding to the narrower frame so that it matches the width of the wider frame, or simply let the stacked result be the sum of the two widths. For most thumbnail use cases, keeping the height uniform and allowing variable width is the cleanest solution.
hstack. Once you have isolated the specific moments you want to showcase, you can proceed with the compositing logic described below.
Step-by-Step Implementation
The following command demonstrates how to extract a frame from a portrait video and a landscape video, resize them to a standard height, pad the portrait video to match the landscape width, and finally merge them. In this example, we target a standard output height of 360 pixels, which corresponds to the 360p resolution often used for web previews.
ffmpeg -i portrait.mp4 -i landscape.mp4 \
-filter_complex "[0:v]scale=-1:360,pad=width=ih*16/9:height=360:x=(ow-iw)/2:y=0:black[v0]; [1:v]scale=-1:360[v1]; [v0][v1]hstack" \
frames.jpgLet us break down the -filter_complex argument, as this is where the magic happens. The filter chain operates on three distinct stages: scaling, padding, and stacking.
1. Scaling to Common Height
The scale=-1:360 instruction tells FFmpeg to set the height to exactly 360 pixels. The width is calculated automatically (-1) to maintain the original aspect ratio. This ensures that the portrait clip becomes narrow but tall enough, and the landscape clip becomes wide but equally tall. Without this step, the hstack filter would reject the inputs due to height mismatch.
2. Padding for Alignment
Although hstack does not strictly require equal widths, padding improves visual consistency. In the example above, we assume the landscape video is the reference for width. However, a more robust method is to calculate the maximum width required. Since landscape is 16:9 and portrait is 9:16, at 360px height, the landscape width is 640px and the portrait width is 240px. To center the portrait within a 640px box, we use pad=width=640:height=360:x=(ow-iw)/2:y=0:black. This adds black bars to the left and right of the portrait video, making it appear as a centered element rather than a tiny strip.
If you do not wish to pad and just want them side-by-side regardless of width differences, you can skip the pad filter entirely. The hstack filter will still work as long as the heights are equal. The resulting image will simply be 880 pixels wide (640 + 240).
3. Horizontal Stacking
The final part, [v0][v1]hstack, takes the processed portrait stream labeled v0 and the landscape stream labeled v1 and places them next to each other. The labels [v0] and [v1] are crucial; they allow us to manipulate the streams independently before merging them. The order determines the layout: the first label appears on the left, and the second on the right.
scale=-1:360 uses bicubic interpolation by default, which is good for general video, you might prefer scale2ref or specific flags if you need pixel-perfect sharpness for text-heavy thumbnails, though bicubic is usually sufficient for natural imagery.
Customizing Dimensions and Output
You can adjust the target height depending on your needs. For high-resolution displays, try 720 or 1080 pixels. Remember that increasing the height increases the file size and processing time linearly. If you are generating many thumbnails, consider using a lower height like 200 pixels for speed.
Additionally, you can control the background color. In the pad filter, the last argument specifies the fill color. Changing black to white or a hex code like #FF0000 allows you to match your brand identity. This is particularly useful if the portrait video has transparent areas or if you want the composite image to stand out against a white webpage background.
Automating the Process
For production environments, you likely need to process multiple pairs of videos. You can wrap this command in a bash script or a Python loop. Ensure that your input filenames correspond correctly to the indices 0:v and 1:v. If you have a list of portrait videos and a separate list of landscape videos, you may need to synchronize them by name or index before feeding them into the FFmpeg command. Always verify the output image visually to ensure the framing captures the key action in both clips.
Frequently Asked Questions
Why does my hstack command fail with "inputs must be of same size"?
This error occurs because the hstack filter strictly requires all input streams to have the exact same height. You must use the scale filter to force both portrait and landscape videos to a common pixel height before attempting to stack them horizontally.
How can I add padding to the portrait video instead of the landscape?
To pad the portrait video, calculate the target width based on the landscape aspect ratio (16:9). Use the pad filter with width=iw*16/9 (if starting from landscape) or a fixed width value, and center it using x=(ow-iw)/2. This ensures the portrait content remains centered within the expanded canvas.
Can I stack videos vertically instead of horizontally?
Yes, replace the hstack filter with vstack. This will place the portrait video on top of the landscape video. Note that you will need to ensure both inputs have the same width, not height, for vertical stacking to work correctly.
What is the best resolution for social media thumbnails?
For platforms like Instagram or TikTok, a height of 360 to 720 pixels is often sufficient. For YouTube or desktop websites, consider 720 or 1080 pixels. Always test the output on the actual device to ensure readability and visual appeal.
How do I handle videos with different frame rates?
FFmpeg handles different frame rates gracefully during filtering. However, for consistent playback if you were converting this to a video, you should use the setpts filter to synchronize timestamps. For static image extraction, frame rate differences do not affect the final composite image quality.