Object Subsystem (ObjectNode)¶
The object subsystem provides S3-compatible object storage interface. It makes possible for ChubaoFS becomes a fusion storage that expose two type interface (POSIX and S3-compatible). So that user can operate files stored in ChubaoFS by using native Amazon S3 SDKs.
Structure¶
The ObjectNode is a functional subsystem node. It fetch volume view (volume topology) on demand from Resource Manager (Master). Each ObjectNode communicate with metadata subsystem (MetaNode) and data subsystem (DataNode) directly.
ObjectNode is stateless design with high scalability and it have ability to operate all files stored in the ChubaoFS cluster which it belong to directly without any volume-mount operation.
Features¶
- Provides object storage interface compatible with native Amazon S3 SDKs.
- Fusion storage expose two type interface (POSIX and S3-compatible).
- Stateless and high scalability
Semantic Transform¶
Based on our POSIX-compatible design. Every file operate request comes from object storage interface need to be made semantic transform to POSIX.
POSIX | Object Storage |
---|---|
Cluster |
Region |
Volume |
Bucket |
Path |
Key |
Example:
Put object ‘example/a/b.txt’ will be create and write data to file ‘/a/b.txt’ in volume ‘example’.
User¶
Before using the object storage service, you need to create a user through the Master. While creating users, AccessKey and SecretKey will be generated for each user.
ChubaoFS uses the field Owner of the volume as the user ID. There are two ways to create users:
- When creating a volume through the API of the Master, if there is no user with the same name as the owner of the volume in the cluster, a user with the user ID of Owner will be automatically created.
- Create a user by calling the user management API of the Master.
Authentication¶
The signature validation algorithm in object storage interface is fully compatible with native Amazon S3 service. The authentication consisting of AccessKey and SecretKey generated by Resource Manager (Master) with user creation, which can be obtained through the Master API. The AccessKey is a 16-character string unique in the entire ChubaoFS cluster.
The user has all access permissions to the volume owned by him. Users can grant other users specified permissions to access volumes under their own names. The permissions are divided into the following three categories:
- Readonly or readwrite permission.
- Permission for a single operation, such as GetObject, PutObject, etc.
- Custom permission.
When a user uses the object storage service to execute a certain operation, ChubaoFS will identify whether the user has the corresponding permission.
Invisible Temporary Data¶
In order to make write operation in object storage interface atomically. Every write operation will create and write data to an invisible temporary. The volume operator in ObjectNode puts file data into temporary which only have ‘inode’ without ‘dentry’ in metadata. When all the file data stored successfully, the volume operator create or update ‘dentry’ in metadata makes it visible to users.
Object Mode Conflict (Important)¶
The POSIX and object storage are two different types of storage product, and the object storage is a Key-Value pair storage service. So that the object with key ‘a/b/c’ and the object with key ‘a/b’ are different object without any conflict.
But ChubaoFS is based on POSIX design. According to semantic transformation rule, the ‘b’ part in key ‘a/b/c’ transform to folder ‘b’ under the folder ‘a’ , and in key ‘a/b’ transform to file ‘b’ under the folder ‘a’.
The object key like this is conflict in ChubaoFS.
When the object being operated has conflicts with the existing object due to the above mode inconsistency, the ObjectNode will return a 409
status code to the client.
Supported S3 Features¶
- File object operations.
- Directory object operations.
- Multipart upload.
- Parallel download for high-level SDK APIs.
- Tagging for bucket and object.
- User-defined metadata for object.
- IP address and network segment black and white list for bucket ACL.
- Signature Algorithm V2 and V4.
- Cross-Origin Resource Sharing (CORS).
Unsupported S3 Features¶
- Version
- Restore deleted objects
- Locking objects
- Lifecycle configuration for bucket and object.
- Hosting Websites
- Encryption
- BitTorrent
Supported APIs¶
Supported SDKs¶
Object Node provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.
Name | Language | Link |
---|---|---|
AWS SDK for Java | Java |
https://aws.amazon.com/sdk-for-java/ |
AWS SDK for JavaScript | JavaScript |
https://aws.amazon.com/sdk-for-browser/ |
AWS SDK for JavaScript in Node.js | JavaScript |
https://aws.amazon.com/sdk-for-node-js/ |
AWS SDK for Go | Go |
https://docs.aws.amazon.com/sdk-for-go/ |
AWS SDK for PHP | PHP |
https://aws.amazon.com/sdk-for-php/ |
AWS SDK for Ruby | Ruby |
https://aws.amazon.com/sdk-for-ruby/ |
AWS SDK for .NET | .NET |
https://aws.amazon.com/sdk-for-net/ |
AWS SDK for C++ | C++ |
https://aws.amazon.com/sdk-for-cpp/ |
Boto3 | Python |
http://boto.cloudhackers.com |