Introduction
SeaweedFS is a straightforward and extremely scalable distributed file system. There are two goals:
- to retailer billions of information!
- to serve the information quick!
SeaweedFS began as an Object Retailer to deal with small information effectively. As an alternative of managing all file metadata in a central grasp, the central grasp solely manages volumes on quantity servers, which handle information and their metadata. This relieves concurrency strain from the central grasp and spreads file metadata into quantity servers, permitting quicker file entry (O(1), normally only one disk learn operation).
There’s solely 40 bytes of disk storage overhead for every file’s metadata. It’s so easy with O(1) disk reads that you’re welcome to problem the efficiency together with your precise use instances.
SeaweedFS began by implementing Fb’s Haystack design paper. Additionally, SeaweedFS implements erasure coding with concepts from f4: Fb’s Heat BLOB Storage System and has plenty of similarities with Fb’s Tectonic Filesystem
On high of the article retailer, the optionally available Filer can help directories and POSIX attributes. Filer is a separate linearly-scalable stateless server with customizable metadata shops, e.g., MySql, Postgres, Redis, Cassandra, HBase, MongoDB, Elastic Search, LevelDB, RocksDB, Sqlite, MemSql, TiDB, Etcd, CockroachDB, YDB, and so on.
For any distributed key-value shops, the big values will be offloaded to SeaweedFS. With the quick entry pace and linearly scalable capability, SeaweedFS can work as a distributed Key-Giant-Worth retailer.
SeaweedFS can transparently combine with the cloud. With scorching knowledge on the native cluster and heat knowledge on the cloud with O(1) entry time, SeaweedFS can obtain each quick native entry time and elastic cloud storage capability. What’s extra, the cloud storage entry API price is minimized. Extra speedy and Cheaper than direct cloud storage!
Further Options
- Can select no replication or completely different replication ranges, rack and knowledge middle conscious.
- Computerized grasp servers failover – no single level of failure (SPOF).
- Computerized compaction to reclaim disk house after deletion or replace.
- Computerized entry TTL expiration.
- Any server with some disk house can add to the overall cupboard space.
- Including/Eradicating servers does not trigger any knowledge re-balancing until triggered by admin instructions.
- Computerized Gzip compression is determined by file MIME sort.
- Non-compulsory image resizing.
- Assist ETag, Settle for-Vary, Final-Modified, and so on.
- Assist in-memory/leveldb/read-only mode tuning for reminiscence/efficiency stability.
- Assist rebalancing the writable and read-only volumes.
- Customizable A number of Storage Tiers: Customizable storage disk varieties to stability efficiency and price.
- Clear cloud integration: limitless capability by way of tiered cloud storage for heat knowledge.
- Erasure Coding for heat storage Rack-Conscious 10.4 erasure coding reduces storage price and will increase availability.
Filer Options
Kubernetes
Instance: Utilizing Seaweed Object Retailer
By default, the grasp node runs on port 9333, and the amount nodes run on port 8080. Let’s begin one grasp node, and two quantity nodes on port 8080 and 8081. Ideally, they need to be began from completely different machines. We’ll use localhost for example.
SeaweedFS makes use of HTTP REST operations to learn, write, and delete. The responses are in JSON or JSONP format.
Begin Grasp Server
