A Golang-based distributed POSIX file system

JuiceFS is a high-performance POSIX file system launched below Apache License 2.0, primarily designed for the cloud-native surroundings. The info, saved by way of JuiceFS, shall be persevered in object storage (e.g. Amazon S3), and the corresponding metadata will be persevered in numerous database engines akin to Redis, MySQL, and TiKV primarily based on the situations and necessities.

With JuiceFS, large cloud storage will be straight related to large knowledge, machine studying, synthetic intelligence, and numerous utility platforms in manufacturing environments. With out modifying code, the large cloud storage can be utilized as effectively as native storage.

 Doc: Fast Begin Information

Highlighted Options

1. Absolutely POSIX-compatible: Use as an area file system, seamlessly docking with present functions with out breaking enterprise workflow.
2. Absolutely Hadoop-compatible: JuiceFS’ Hadoop Java SDK is suitable with Hadoop 2.x and Hadoop 3.x in addition to numerous parts within the Hadoop ecosystems.
3. S3-compatible: JuiceFS’ S3 Gateway offers an S3-compatible interface.
4. Cloud Native: A Kubernetes CSI Driver is offered for simply utilizing JuiceFS in Kubernetes.
5. Shareable: JuiceFS is a shared file storage that may be learn and written by 1000’s of purchasers.
6. Robust Consistency: The confirmed modification shall be instantly seen on all of the servers mounted with the identical file system.
7. Excellent Efficiency: The latency will be as little as just a few milliseconds, and the throughput will be expanded almost unlimitedly (relying on the scale of the item storage). Check outcomes
8. Knowledge Encryption: Helps knowledge encryption in transit and at relaxation (please confer with the information for extra data).
9. International File Locks: JuiceFS helps each BSD locks (flock) and POSIX document locks (fcntl).
10. Knowledge Compression: JuiceFS helps LZ4 or Zstandard to compress all of your knowledge.

Structure

JuiceFS can retailer the metadata of file system on Redis, which is a quick, open-source, in-memory key-value knowledge storage, notably appropriate for storing metadata; in the meantime, all the information shall be saved in object storage via JuiceFS consumer. Be taught extra

Every file saved in JuiceFS is break up into “Chunk” s at a hard and fast measurement with the default higher restrict of 64 MiB. Every Chunk consists of a number of “Slice”(s), and the size of the slice varies relying on how the file is written. Every slice consists of size-fixed “Block” s, that are 4 MiB by default. These blocks shall be saved in object storage ultimately; on the identical time, the metadata data of the file and its Chunks, Slices, and Blocks shall be saved in metadata engines by way of JuiceFS. Be taught extra

When utilizing JuiceFS, recordsdata will ultimately be break up into Chunks, Slices and Blocks and saved in object storage. Due to this fact, the supply recordsdata saved in JuiceFS can’t be discovered within the file browser of the item storage platform; as a substitute, there are solely a chunks listing and a bunch of digitally numbered directories and recordsdata within the bucket. Don’t panic! That is simply the key of the high-performance operation of JuiceFS!

Getting Began

1. Redis database for metadata storage
2. Object storage for storing knowledge blocks
3. JuiceFS Consumer downloaded and put in

Please confer with Fast Begin Information to begin utilizing JuiceFS straight away!

Command Reference

Take a look at all of the command line choices in command reference.

Kubernetes

It is usually very straightforward to make use of JuiceFS on Kubernetes. Please discover extra data right here.

Hadoop Java SDK

When you wanna use JuiceFS in Hadoop, verify Hadoop Java SDK.

Superior Subjects

Please confer with JuiceFS Doc Heart for extra data.

POSIX Compatibility

JuiceFS has handed all compatibility exams (8813 in whole) within the newest pjdfstest .

All exams profitable.

Check Abstract Report
-------------------
/root/tender/pjdfstest/exams/chown/00.t          (Wstat: 0 Checks: 1323 Failed: 0)
  TODO handed:   693, 697, 708-709, 714-715, 729, 733
Information=235, Checks=8813, 233 wallclock secs ( 2.77 usr  0.38 sys +  2.57 cusr  3.93 csys =  9.65 CPU)
Consequence: PASS

Other than the POSIX options lined by pjdfstest, JuiceFS additionally offers:

• Shut-to-open consistency. As soon as a file is written and closed, it’s assured to view the written knowledge within the following open and browse. All of the written knowledge will be learn instantly throughout the identical mount level.
• Rename and all different metadata operations are atomic, assured by Redis transaction.
• Opened recordsdata stay accessible after unlink from identical mount level.
• Mmap (examined with FSx).
• Fallocate with punch gap help.
• Prolonged attributes (xattr).
• BSD locks (flock).
• POSIX document locks (fcntl).

Efficiency Benchmark

Fundamental benchmark

Throughput

A sequential learn/write benchmark has additionally been carried out on JuiceFS, EFS and S3FS by fio.

The determine above reveals that JuiceFS can present 10X extra throughput than the opposite two (see extra particulars).

Metadata IOPS

A easy mdtest benchmark has been carried out on JuiceFS, EFS and S3FS by mdtest.

The end result reveals that JuiceFS can present considerably extra metadata IOPS than the opposite two (see extra particulars).

Analyze efficiency

$ cat /jfs/.accesslog
2021.01.15 08:26:11.003330 [uid:0,gid:0,pid:4403] write (17669,8666,4993160): OK <0.000010>
2021.01.15 08:26:11.003473 [uid:0,gid:0,pid:4403] write (17675,198,997439): OK <0.000014>
2021.01.15 08:26:11.003616 [uid:0,gid:0,pid:4403] write (17666,390,951582): OK <0.000006>

The final quantity on every line is the time (in seconds) that the present operation takes. You possibly can straight use this to debug and analyze efficiency points, or strive juicefs profile /jfs to watch actual time statistics. Please run juicefs profile -h or confer with right here to study extra about this subcommand.

Supported Object Storage

JuiceFS helps nearly all object storage providers. Be taught extra.

Who’s utilizing

JuiceFS is manufacturing prepared and utilized by 1000’s of machines in manufacturing. A listing of customers has been assembled and documented right here. As well as, JuiceFS has a number of collaborative tasks that combine with different open-source tasks, which we’ve documented right here. If you’re additionally utilizing JuiceFS, please be happy to tell us, and you might be welcome to share your particular expertise with everybody.

The storage format is secure, and shall be supported by all future releases.

Roadmap

Reporting Points

We use GitHub Points to trace group reported points. You too can contact the group for any questions.

Contributing

Thanks to your contribution! Please confer with the JuiceFS Contributing Information for extra data.

Group

Welcome to affix the Discussions and the Slack channel to attach with JuiceFS workforce members and different customers.

Utilization Monitoring

JuiceFS collects nameless utilization knowledge by default to assist us higher perceive how the group is utilizing JuiceFS. Solely core metrics (e.g. model quantity) shall be reported, and consumer knowledge and some other delicate knowledge won’t be included. The associated code will be considered right here.

You could possibly additionally disable reporting simply by command line choice --no-usage-report:

juicefs mount --no-usage-report

License

JuiceFS is open-sourced below Apache License 2.0, see LICENSE.

Credit

The design of JuiceFS was impressed by Google File System, HDFS and MooseFS. Thanks for his or her wonderful work!