AIOZ Network’s File Storage Distribution Optimization
Improving AIOZ Network is a technical endeavor that requires innovative approaches to create solutions that allow it to scale and handle large amounts of data while remaining decentralized.
HLS and MPEG-DASH video streaming protocols have become so popular that they have become ubiquitous on almost any online video streaming platform. Both protocols have one thing in common: the videos are required to be segmented into many small chunks with a duration of a few seconds.
For example, a 2-hour movie converted into four profiles (480p, 720p, 1080p, 4k) takes up thousands of small chunks. With the massive uploaded videos nowadays, these small chunks will quickly explode to billions or even trillions.
That amount of workload is not suitable for the decentralized environment of blockchain. The consequence is that it degrades the performance of the whole network over time, resulting in failed uploading, long finalization time, and languishing video buffering. Therefore, we propose the Archive Structure, which reduces the workload by a factor of thousands.
1. Archive Structure
From the perspective of AIOZ Network architecture, the smallest storage unit is a file. There is no lower bound of file size. However, to optimize the workloads and performance of the AIOZ Network, the file size is recommended to be extensive (gigabytes). There will be a floor price applying to small files to encourage users to store large files.
The case of storing many small files can be covered so that the file format is an archive format that can contain many internal small files (tar, zip, rar). Accessing the internal files can be done by reading a range of bytes of the large file.
All of this logic is implemented at the clients’ side to read the internal files. AIOZ Network’s nodes do not maintain any file format metadata, which delivers the requested range of bytes by the clients.
To implement the technique, the archive format has to meet a few requirements:
- The archive maintains a small index at the beginning or the end of the file.
- The internal files are stored continuously and initially inside the archive.
- The index provides the size and location of the internal files.
For those requirements, we choose the ZIP format to be the built-in archive format in AIOZ Network. This technique comes with one condition: that the zip file only stores files without compression. We will have more archive formats if the ZIP format is no longer suitable in the future.
ZIP Internal layout
Reading internal files is a 2-step process:
- The client requests a few kilobytes at the end of the archive to get the internal files index because the ZIP archive stores the index at the end of the library.
- The client requests the bytes range corresponding to the internal file needed to read. The client may repeat this step to read many internal files.
2. Applying to video streaming
The standard video streaming protocols for the web (HLS, MPEG-DASH) require the videos to be divided into many tiny segments, usually just a few seconds per segment, to be instantly delivered via HTTP(s). Such protocols are well suited to the archive format above.
For example, this is the layout of an archive that packages an HLS video stream. This approach can be applied to MPEG-DASH similarly.
Layout of a ZIP archive that packages an HLS video stream.
The content of master_playlist.m3u8:
The content of 720p_playlist.m3u8:
Notice that all the links to the internal files have range=<offset>,<size>. With that information, the clients know the exact bytes range to request the needed files.
The clients could perform the following steps to play the HLS video:
- The client requests a few kilobytes at the end of the archive to get the internal files index. Because the master_playlist.m3u8 is tiny (hundreds of bytes), it is usually included in the first response. Otherwise, the clients may request again with a bigger size.
- With the master_playlist.m3u8, the clients know where the 720p_playlist.m3u8 and 1080p_playlist.m3u8 are located in the archive. The clients request the specific byte range to get the 720p_playlist.m3u8, assuming the 720p profile is selected to play.
- With the 720p_playlist.m3u8, now the clients can request the media segments to play.
This approach also can support subtitles, thumbnail images with variant format (jpeg for static, webp for animated thumbnails), cover images, or anything that comes with the video.
With many such unique optimizations later, AIOZ Network can handle enormous amounts of videos and data inside the ecosystem.
About the AIOZ Network
The AIOZ Network is a Layer-1 Blockchain-based Content Delivery Network that is about to bring a revolution to the entertainment industry.
AIOZ Network utilizes Blockchain for better content distribution by leveraging the power of decentralization. Unlike the traditional data centers operating on a centralized model. distributed Content Delivery Network (dCDN) uses Nodes for storing, streaming, and transferring data.
The AIOZ Network uses a faster, cheaper, and more robust infrastructure for content streaming making it more affordable, faster. This allows the AIOZ Network to provide a higher quality service.
By using this revolutionary technology, the AIOZ Network can efficiently change the way the world streams content of all sorts, improving the quality of life of many and shaping the re-evolution of information and knowledge for future generations.
