![]() ![]() It’s basically a glorified version of Put, Get, and Delete. Within a region things can get quite complicated, but the interface between regions is quite simple. Magic Pocket stores blocks in a highly reliable manner within a storage region but it also stores this data independently in at least two separate regions. To understand our cold storage developments we first need a high level understanding of how Magic Pocket works. Replicating data across geographical regions makes Magic Pocket resilient to large scale natural disasters but also significantly less efficient in aggregate. Within a given geographic region Magic Pocket is already a highly efficient storage system. When we first upload a file to Magic Pocket we use n-way replication across a relatively large number of storage nodes, but then later encode older data in a more efficient erasure coded format in the background. Magic Pocket also uses different data encodings as files age. We save the solid-state drives (SSDs) for our databases and caches. It uses spinning disks, which have the advantage of being cheap, durable, and relatively high-bandwidth. This system is already designed for a fairly cold workload. Magic Pocket’s job is to durably store and serve those large blocks. The blocks are immutable-all metadata operations and related complexities around mutations and revision history are handled by the metadata layers on top. ![]() Dropbox splits files into chunks, called blocks, up to 4MB in size. ![]() Magic Pocket is our system for storing files. We refer to data that’s accessed frequently as “warm” and infrequently accessed data as “cold.” The differences in access characteristics between warm and cold data open up opportunities for cost optimization by tailoring the system to each class of data. In general, people are much more likely to access files they have recently uploaded rather than files they uploaded years ago. Users also tend to share new documents, so a file is also likely to be synced to other devices soon after upload. A new upload triggers a number of internal systems that fetch the file in order to augment the user experience, such as perform OCR, parse content to extract search tokens, or generate web previews for Office documents. Over 40% of all file retrievals in Dropbox are for data uploaded in the last day, over 70% for data uploaded in the last month, and over 90% for data uploaded in the last year. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |