How to Send Multi Video in UDP Protocol with Single IP/Port?

Understanding UDP Limitations

UDP (User Datagram Protocol) operates at Layer 4 of the OSI model. Unlike TCP, which establishes a virtual circuit allowing for ordered, reliable data transfer between two endpoints, UDP simply fires packets onto the network. A single UDP socket is bound to one local IP address and one local port number. When a receiver listens on :8080, it receives raw bytes. It has no inherent mechanism to know if those bytes belong to "Video Stream A" or "Video Stream B" unless the application layer defines a structure.

If you attempt to bind two separate FFmpeg processes to the exact same ip:port combination on the sender side, the operating system will reject the second binding. If you try to send two different streams to the same destination ip:port from different sources, the receiver will receive a chaotic mix of interleaved packets, making it impossible to decode either stream correctly without complex demultiplexing logic.

Multiplexing via Container Formats (MPEG-TS)

The most robust way to send multiple videos over a single UDP pipe is to combine them into a single transport stream before transmission. The MPEG-2 Transport Stream (TS) format is designed specifically for this purpose. It allows multiple Program Streams (each containing video, audio, and subtitles) to be multiplexed into a single elementary stream identified by a Program Map Table (PMT).

1. Encode each video separately: Prepare your source files. For example, let's assume you have video1.mp4 and video2.mp4.
2. Use FFmpeg to remux into TS: You can feed both inputs into a single FFmpeg command that outputs a single MPEG-TS file. This file contains PID (Packet Identifier) tags that allow the receiver to separate the streams later.

ffmpeg -i video1.mp4 -i video2.mp4 \
-c copy \
-map 0:v -map 1:v \
-f mpegts udp://239.255.1.1:1234?pkt_size=1316

In this command, -c copy ensures no re-encoding occurs, preserving quality and reducing CPU load. The -f mpegts flag forces the output format to MPEG-TS. The destination URL uses a multicast IP (239.255.1.1) which is standard for broadcasting multiple streams, though unicast IPs work similarly. The ?pkt_size=1316 parameter ensures the UDP packets fit within standard Ethernet MTU sizes, preventing fragmentation issues.

Using RTP with SSRC Multiplexing

If you need to keep the streams separate but share the same port, you can use Real-time Transport Protocol (RTP) over UDP. RTP includes a header field called SSRC (Synchronization Source Identifier). Each stream must be assigned a unique SSRC. While they share the same destination IP and Port, the receiving application must parse the RTP headers to distinguish between the streams.

This approach is more complex because standard media players often expect a single stream per port. You typically need a custom receiver or a specialized server (like GStreamer or a custom C/C++ application) to handle the demuxing based on SSRC values. FFmpeg can generate these streams, but consuming them requires careful configuration.

# Example for Stream 1
ffmpeg -i input1.mp4 -c:v libx264 -an -f rtp rtp://239.255.1.1:1234?ssrc=100

# Example for Stream 2
ffmpeg -i input2.mp4 -c:v libx264 -an -f rtp rtp://239.255.1.1:1234?ssrc=200

Note that many simple UDP receivers will drop packets that don't match their expected sequence numbers or SSRCs. Therefore, while technically feasible, this method is less compatible than the MPEG-TS approach.

Alternative: Separate Ports

If multiplexing is not required and you simply want to send multiple videos efficiently, the standard practice is to assign a unique UDP port to each stream. For example, Video 1 goes to :1234 and Video 2 goes to :1235. This avoids all complexity related to packet identification and is supported by virtually all networking hardware and software players.

However, if firewall rules restrict the number of open ports, the MPEG-TS solution described above is the best compromise. It allows unlimited streams to pass through a single port entry point.

Handling Packet Loss

UDP does not guarantee delivery. When sending multiple high-bitrate videos, packet loss can cause severe artifacts. Consider adding Forward Error Correction (FEC) if your protocol supports it, or switching to a protocol like QUIC or WebRTC if reliability is critical. For pure UDP, ensure your buffer sizes are large enough to absorb minor jitter.

Frequently Asked Questions

Can I use TCP instead of UDP for multiple streams?

Yes, TCP is inherently better for multiple streams because it manages connections separately. However, TCP introduces latency due to handshake and retransmission overhead, which may be unacceptable for real-time live streaming applications where low latency is prioritized over perfect accuracy.

What happens if I ignore the SSRC in an RTP stream?

If you ignore the SSRC, the receiver will treat all incoming packets as part of a single stream. This results in corrupted video frames, constant seeking errors, and complete playback failure because the decoder cannot synchronize the keyframes properly.

Is multicast required for sending multiple videos over one port?

No, multicast is not strictly required, but it is highly recommended for multi-stream scenarios. Multicast allows efficient bandwidth usage when multiple receivers subscribe to the same content. Unicast works fine for point-to-point transmission but consumes more bandwidth if multiple clients request the same data.

How do I debug why my UDP stream is dropping packets?

Check your network interface statistics for dropped packets. Use tools like Wireshark to analyze the payload. Ensure your pkt_size matches your network's MTU. Also, verify that your OS socket buffer sizes are sufficient for the bitrate of the combined streams.

Can I concatenate images and videos in the same UDP stream?

Not directly in a standard video stream. Images require decoding into frames first. However, you can include image sequences as separate tracks within an MPEG-TS container alongside video tracks, similar to how you would handle multiple video streams. See our guide on concatenating image and video without reencoding for related techniques.