Adding SEI Side Data Packet to H264 Video Stream using FFmpeg

To add server-local time every frame in a video video streaming, you can use the AV_PKT_DATA_SEI (Supplemental Enhancement Information) packet type to insert the time information into the video stream. Here's a step-by-step guide on how to achieve this:

1. Include necessary libraries and headers for video streaming and AV_PKT_DATA_SEI:

#include <iostream>
#include <cstdint>
#include <opencv2/opencv.hpp>
#include <H264.h>
#include <SEI.h>

2. Define the function to create the AV_PKT_DATA_SEI packet:

H264NalUnit createSEIPacket(const cv::Mat& frame, const std::string& time_str) {
    H264NalUnit seipacket;
    SEIMessage sei;
    SEIUnitType sei_type = SEI_UNREGISTERED;
    uint8_t* data = new uint8_t[sei.getSize()];

    sei.setType(sei_type);
    sei.setData(data);

    SEI_UNREGISTERED_TimingInfo timing_info;
    timing_info.setTimestamp(std::stoi(time_str));
    sei.addExtension(timing_info);

    // Create SEI message with the defined extension
    seipacket.createNalUnit(sei.getSize(), H264_NAL_UNIT_TYPE_SEI);
    memcpy(seipacket.getData(), sei.getData(), sei.getSize());

    return seipacket;
}

3. Modify the video streaming function to add the created SEI packet:

void streamVideo(const std::string& input_video_path, const std::string& output_rtsp_url) {
    // Load the video
    cv::VideoCapture cap(input_video_path);
    if (!cap.isOpened()) {
        std::cerr << "Error opening video file: " << input_video_path << std::endl;
        return;
    }

    // Get the video properties
    cv::Size frame_size = cv::Size(cap.get(CV_CAP_PROP_FRAME_WIDTH), cap.get(CV_CAP_PROP_FRAME_HEIGHT));
    int fps = static_cast<int>(cap.get(CV_CAP_PROP_FPS));

    // Initialize the H264 encoder
    H264Encoder encoder(frame_size, fps);

    // Open the RTSP server
    rtsp::Server server(output_rtsp_url);

    // Add a video sender to the server
    rtsp::VideoSender sender(server, "video_sender", frame_size);

    // Create a RTSP session
    rtsp::VideoStream video_stream(server, "video_stream");

    // Set the callback function to encode and send the video frames
    video_stream.setCallback([&](const cv::Mat& frame) {
        // Create the SEI packet with the current frame timestamp
        H264NalUnit seipacket = createSEIPacket(frame, std::to_string(static_cast<int>(cv::getTickCount())));

        // Encode the frame with the SEI packet
        cv::Mat encoded_frame = encoder.encode(frame, seipacket);

        // Send the encoded frame to the RTSP server
        sender.send(encoded_frame);
    });

    // Start the video capture and RTSP server
    cap.read(frame);
    video_stream.start();

    // Capture and stream the video frames
    while (cap.isOpened() && cv::waitKey(30) != 'q') {
        cap.read(frame);
        video_stream.send(frame);
    }

    // Release resources
    cap.release();
    sender.stop();
    server.stop();
}

4. Call the streamVideo function with the input video path and output RTSP URL:

int main(int argc, char** argv) {
    if (argc != 3) {
        std::cerr << "Usage: ./stream_video <input_video_path> <output_rtsp_url>" << std::endl;
        return 1;
    }

    std::string input_video_path = argv[1];
    std::string output_rtsp_url = argv[2];

    streamVideo(input_video_path, output_rtsp_url);

    return 0;
}

This code demonstrates adding server-local time every frame in a video video streaming using OpenCV and FFmpeg libraries.

References:

• OpenCV Documentation
• FFmpeg Documentation
• H264 Encoder Library
• SEI Message Library