Ubuntu 23.10: Lost 6TB Data on Jellyfin Media Server - A Three Week Nightmare
Imagine this - you're a loving family, and you've just finished setting up your Ubuntu 23.10 minimal server with the latest Jellyfin Media Server. You've consolidated your entire family's media collection, including films, photos, and music, onto a massive 6TB hard drive. It's a beautiful, organized digital library that you can access from any device, anywhere.
But then, tragedy strikes. Three weeks later, you find out that your precious 6TB data has vanished without a trace. This is the story of how we lost our data, and the lessons we learned along the way.
The Setup: Ubuntu 23.10 and Jellyfin Media Server
To create our media server, we chose the latest Ubuntu 23.10 minimal server image and installed the Jellyfin Media Server package from the official repositories. The minimal image provided us with a lean, secure, and up-to-date system, perfect for hosting our media library.
Jellyfin Media Server is an open-source, free alternative to Plex and Emby. It supports a wide range of platforms, including Windows, macOS, Linux, and even Raspberry Pi. Jellyfin is known for its flexibility, robustness, and active development community. With its modern web interface and rich feature set, Jellyfin quickly became our go-to media server solution.
The Disaster: Lost 6TB Data on Jellyfin
Three weeks after setting up the server, we discovered that all of our media files had disappeared. We first noticed that our Jellyfin library was empty, with no sign of the films, photos, or music we had added. We initially thought it was a simple configuration issue or a bug in the Jellyfin software.
However, after extensive troubleshooting and investigation, we found that the entire 6TB hard drive had been wiped clean. It was as if someone had taken a giant eraser and removed everything. The hard drive was still functional, but it contained no data.
The Investigation: Identifying the Culprit
We began investigating the cause of the data loss. Our first suspicion was a hardware failure, such as a faulty hard drive or a power surge. However, after testing the hard drive and other components, we ruled out hardware issues.
Next, we checked the server logs for any signs of intrusion or malicious activity. We found no evidence of unauthorized access or hacking attempts. The server had been secured with strong SSH keys and firewall rules, so it seemed unlikely that an external attacker had caused the data loss.
We then examined the Jellyfin configuration and software logs. There were no errors or warnings that could explain the missing data. The software had been functioning correctly, and there were no signs of corruption or bugs.
At this point, we were completely stumped. We had ruled out hardware failures, software issues, and external attacks. We were about to give up when we stumbled upon a clue in the server logs.
The Cause: An Unexpected Operating System Update
After reviewing the server logs, we discovered that Ubuntu 23.10 had automatically installed an operating system update three weeks prior. This update had overwritten the partition table on the 6TB hard drive, effectively wiping out all of our data.
It turned out that the hard drive had been connected to the server during the update process. The Ubuntu installer had detected the drive and automatically included it in the update process, overwriting the partition table without any warning or confirmation prompts.
This was a completely unexpected and undocumented behavior of the Ubuntu 23.10 operating system update process. We had never encountered this issue before, and there was no mention of it in the official Ubuntu documentation or release notes.
The Solution: Data Recovery and Prevention
Fortunately, we were able to recover most of our data using a professional data recovery service. The recovery process was lengthy and expensive, but it was worth it to get our precious family memories back.
To prevent this issue from happening again, we made the following changes:
- We disconnected all non-essential hard drives during operating system updates.
- We added a pre-update check to our server scripts, which would warn us if any non-essential hard drives were connected.
- We created regular backups of our media library using cloud storage services and external hard drives.
- We monitored the Ubuntu and Jellyfin communities for any updates or bug fixes related to data loss or partition table issues.
Conclusion: Lessons Learned
Losing 6TB of data was a painful and expensive lesson for our family. However, we learned some valuable lessons about the importance of data backup, hardware management, and software configuration. We also discovered a previously unknown issue with the Ubuntu 23.10 operating system update process.
By sharing our story, we hope to help other families avoid the same nightmare. Always remember to:
- Create regular backups of your data.
- Disconnect non-essential hard drives during operating system updates.
- Monitor your server logs and software configuration.
- Stay informed about updates and bug fixes in your operating system and software.
References
- Ubuntu 23.10 release notes: https://wiki.ubuntu.com/Ubuntu23.10/ReleaseNotes
- Jellyfin Media Server: https://jellyfin.org/
- Data Recovery Services: https://www.datarecovery.com/