Welcome to ChubaoFS(Chubao File System)

Introduction

ChubaoFS(Chubao File System) is a distributed file system that is designed to natively support large scale container platforms.

High Level Architecture

ChubaoFS consists of a metadata subsystem, a data subsystem, and a resource manager, and can be accessed by different clients (as a set of application processes) hosted on the containers through different file system instances called volumes.

The metadata subsystem stores the file metadata, and consists of a set of meta nodes. Each meta node consists of a set of meta partitions.

The data subsystem stores the file contents, and consists of a set of data nodes. Each data node consists of a set of data partitions.

The volume is a logical concept in ChubaoFS and consists of one or multiple meta partitions and one or multiple data partitions. Each partition can only be assigned to a single volume. From a client’s perspective, the volume can be viewed as a file system instance that contains data accessible by the containers. A volume can be mounted to multiple containers so that files can be shared among different clients simultaneously, and needs to be created at the very beginning before the any file operation. A ChubaoFS cluster deployed at each data center can have hundreds of thousands of volumes, whose data sizes vary from a few gigabytes to several terabytes.

Generally speaking, the resource manager periodically communicates with the metadata subsystem and data subsystem to manage the meta nodes and data nodes, respectively. Each client periodically communicates with the resource manager to obtain the up-to-date view of the mounted volume. A file operation usually initiates the communications from the client to the corresponding meta node and data node directly, without the involvement of the resource manager. The updated view of the mounted volume, as well as the file metadata are usually cached at the client side to reduce the communication overhead.

Features

Scalable Meta Management

The metadata operations could make up as much as half of typical file system workloads. On our platform, this becomes even more important as there could be heavy accesses to the metadata of files by tens of thousands of clients simultaneously. A single node that stores the file metadata could easily become the performance bottleneck. As a result, we employ a distributed metadata subsystem to manage the file metadata. In this way, the metadata requests from different clients can be forwarded to different nodes, which improves the scalability of the entire system. The metadata subsystem can be considered as an in-memory datastore of the file metadata. It can have thousands of meta nodes, each of which can have hundreds of billions of meta partitions. Each meta partition on a meta node stores the file metadata in memory by maintaining a set of inodes and a set of dentries. We employ two b-trees called inodeTree and dentryTree for fast lookup of inodes and dentries in the memory. The inodeTree is indexed by the inode id, and the dentryTree is indexed by the dentry name and the parent inode id. We also maintain a range of the inode ids (denoted as start and end) stored on a meta partition for splitting.

General-Purpose Storage Engine

To reduce the storage cost, many applications and services are served from the same shared storage infrastructure (aka “multi-tenancy”). The workloads of different applications and services are mixed together, where the file size can vary from a few kilobytes to hundreds of gigabytes, and the files can be written in a sequential or random fashion. For example, the log files usually need to be written sequentially in the execution order of the code; some data analytics in the machine learning domain are based on the data stored on the underlying file system; and a database engine running on top of the file system can modify the stored data frequently. A dedicated file system needs to be able to serve for all these different workloads with excellent performance.

Strong Replication Consistency

E-commence venders who move their line of business applications to the cloud usually prefer strong consistency. For example, an image processing service may not want to provide the customer with an outdated image that does not match the product description. This can be easily achieved if there is only one copy of the file. But to ensure a distributed file system to continue operating properly in the event of machines failures, which can be caused by various reasons such as faulty hard drives, bad motherboards, etc, there are usually multiple replicas of the same file. As a result, in a desired file system, the data read from any of the replicas must be consistent with each other.

Relaxed POSIX Semantics and Metadata Atomicity

In a POSIX-compliant distributed file system, the behavior of serving multiple processes on multiple client nodes should be the same as the behavior of a local file system serving multiple processes on a single node with direct attached storage. ChubaoFS provides POSIX-compliant APIs. However, the POSIX consistency semantics, as well as the atomicity requirement between the inode and dentry of the same file, have been carefully relaxed in order to better align with the needs of applications and to improve the system performance.

Run Cluster by Yum Tools

Yum tools to run a ChubaoFS cluster for CentOS 7+ is provided. The list of RPM packages dependencies can be installed with:

$ yum install http://storage.jd.com/chubaofsrpm/latest/cfs-install-latest-el7.x86_64.rpm
$ cd /cfs/install
$ tree -L 2
.
├── install_cfs.yml
├── install.sh
├── iplist
├── src
└── template
    ├── client.json.j2
    ├── console.json.j2
    ├── create_vol.sh.j2
    ├── datanode.json.j2
    ├── grafana
    │   ├── grafana.ini
    │   ├── init.sh
    │   └── provisioning
    ├── master.json.j2
    ├── metanode.json.j2
    └── objectnode.json.j2

Set parameters of the ChubaoFS cluster in iplist.

• [master], [datanode] , [metanode] , [console], [monitor] , [client] modules includes member IP addresses of each role.
• [cfs:vars] module includes parameters for SSH connection. So make sure the port, username and password for SSH connection is unified before start.
• #master config module includes parameters of Master.

Properties

Key	Type	Description	Mandatory
master_clusterName	string	The cluster identifier	Yes
master_listen	string	Http port which api service listen on	Yes
master_prof	string	golang pprof port	Yes
master_logDir	string	Path for log file storage	Yes
master_logLevel	string	Level operation for logging. Default is info.	No
master_retainLogs	string	the number of raft logs will be retain.	Yes
master_walDir	string	Path for raft log file storage.	Yes
master_storeDir	string	Path for RocksDB file storage,path must be exist	Yes
master_exporterPort	int	The prometheus exporter port	No
master_metaNodeReservedMem	string	If the metanode memory is below this value, it will be marked as read-only. Unit: byte. 1073741824 by default.	No

• #datanode config module includes parameters of DataNodes.

Properties

Key	Type	Description	Mandatory
datanode_listen	string	Port of TCP network to be listen	Yes
datanode_prof	string	Port of HTTP based prof and api service	Yes
datanode_logDir	string	Path for log file storage	Yes
datanode_logLevel	string	Level operation for logging. Default is info	No
datanode_raftHeartbeat	string	Port of raft heartbeat TCP network to be listen	Yes
datanode_raftReplica	string	Port of raft replicate TCP network to be listen	Yes
datanode_raftDir	string	Path for raft log file storage	No
datanode_exporterPort	string	Port for monitor system	No
datanode_disks	string slice	Format: PATH:RETAIN. PATH: Disk mount point, must exists. RETAIN: Retain space. (Ranges: 20G-50G.) RETAIN: The minimum free space(Bytes) reserved for the path.	Yes

• #metanode config module includes parameters of MetaNodes.

Properties

Key	Type	Description	Mandatory
metanode_listen	string	Listen and accept port of the server	Yes
metanode_prof	string	Pprof port	Yes
metanode_logLevel	string	Level operation for logging. Default is info	No
metanode_metadataDir	string	MetaNode store snapshot directory	Yes
metanode_logDir	string	Log directory	Yes
metanode_raftDir	string	Raft wal directory	Yes
metanode_raftHeartbeatPort	string	Raft heartbeat port	Yes
metanode_raftReplicaPort	string	Raft replicate port	Yes
metanode_exporterPort	string	Port for monitor system	No
metanode_totalMem	string	Max memory metadata used. The value needs to be higher than the value of metaNodeReservedMem in the master configuration. Unit: byte	Yes

• #objectnode config module includes parameters of ObjectNodes.

Key	Type	Description	Mandatory
objectnode_listen	string	Listen and accept port of the server	Yes
objectnode_domains	string slice	Domain of S3-like interface which makes wildcard domain support Format: DOMAIN	No
objectnode_logDir	string	Log directory	Yes
objectnode_logLevel	string	Level operation for logging. Default: error	No
objectnode_exporterPort	string	Port for monitor system	No
objectnode_enableHTTPS	string	Enable HTTPS	Yes

• #console config module includes parameters of Console.

Properties

Key	Type	Description	Mandatory
console_logDir	string	Path for log file storage	Yes
console_logLevel	string	Level operation for logging. Default is info	No
console_listen	string	Port of TCP network to be listen, default is 80	Yes

• #client config module includes parameters of Client.

Properties

Key	Type	Description	Mandatory
client_mountPoint	string	Mount point of the client	Yes
client_volName	string	Volume name	No
client_owner	string	Owner id	Yes
client_SizeGB	string	Initial size to create the volume if it not exists	No
client_logDir	string	Path for log file storage	Yes
client_logLevel	string	Level operation for logging. Default is info	No
client_exporterPort	string	Port exposed to monitor system	Yes
client_profPort	string	Pprof port	No

[master]
10.196.59.198
10.196.59.199
10.196.59.200
[datanode]
...
[cfs:vars]
ansible_ssh_port=22
ansible_ssh_user=root
ansible_ssh_pass="password"
...
#master config
...
#datanode config
...
datanode_disks =  '"/data0:10737418240","/data1:10737418240"'
...
#metanode config
...
metanode_totalMem = "28589934592"
...
#objectnode config
...
#console config
...

For more configurations, please refer to Resource Manager (Master); Meta Subsystem; Data Subsystem; Client; Monitor; Console.

Start the resources of ChubaoFS cluster with script install.sh . (make sure the Master is started first)

$ bash install.sh -h
Usage: install.sh -r | --role [datanode | metanode | master | objectnode | console | monitor | client | all | createvol ]
$ bash install.sh -r master
$ bash install.sh -r metanode
$ bash install.sh -r datanode
$ bash install.sh -r objectnode
$ bash install.sh -r console
$ bash install.sh -r monitor
$ bash install.sh -r client

Check mount point at /cfs/mountpoint on client node defined in iplist .

Open http://consul.prometheus-cfs.local in browser for monitoring system(the IP of monitoring system is defined in iplist ).

Run Cluster Manually

Building

Use following command to build server and client:

make build

If the build is successful, cfs-server and cfs-client will be found in directory build/bin

Deployment

Start Resource Manager (Master)

nohup ./cfs-server -c master.json &

Sample master.json is shown as follows,

{
  "role": "master",
  "ip": "10.196.59.198",
  "listen": "17010",
  "prof":"17020",
  "id":"1",
  "peers": "1:10.196.59.198:17010,2:10.196.59.199:17010,3:10.196.59.200:17010",
  "retainLogs":"2000",
  "logDir": "/cfs/master/log",
  "logLevel":"info",
  "walDir":"/cfs/master/data/wal",
  "storeDir":"/cfs/master/data/store",
  "consulAddr": "http://consul.prometheus-cfs.local",
  "exporterPort": 9500,
  "clusterName":"chubaofs01",
  "metaNodeReservedMem": "1073741824"
}

For detailed explanations of master.json, please refer to Resource Manager (Master).

Start Metanode

nohup ./cfs-server -c meta.json &

Sample meta.json is shown as follows,

{
    "role": "metanode",
    "listen": "17210",
    "prof": "17220",
    "logLevel": "info",
    "metadataDir": "/cfs/metanode/data/meta",
    "logDir": "/cfs/metanode/log",
    "raftDir": "/cfs/metanode/data/raft",
    "raftHeartbeatPort": "17230",
    "raftReplicaPort": "17240",
    "totalMem":  "8589934592",
    "consulAddr": "http://consul.prometheus-cfs.local",
    "exporterPort": 9501,
    "masterAddr": [
        "10.196.59.198:17010",
        "10.196.59.199:17010",
        "10.196.59.200:17010"
    ]
}

For detailed explanations of meta.json, please refer to Meta Subsystem.

Start Datanode

1. Prepare data directories

   Recommendation Using independent disks can reach better performance.

   Disk preparation

   1.1 Check available disks

   fdisk -l
   

   1.2 Build local Linux file system on the selected devices

   mkfs.xfs -f /dev/sdx
   

   1.3 Make mount point

   mkdir /data0
   

   1.4 Mount the device on mount point

   mount /dev/sdx /data0
   

2. Start datanode

   nohup ./cfs-server -c datanode.json &
   

   Sample datanode.json is shown as follows,

   {
     "role": "datanode",
     "listen": "17310",
     "prof": "17320",
     "logDir": "/cfs/datanode/log",
     "raftDir": "/cfs/datanode/log",
     "logLevel": "info",
     "raftHeartbeat": "17330",
     "raftReplica": "17340",
     "consulAddr": "http://consul.prometheus-cfs.local",
     "exporterPort": 9502,
     "masterAddr": [
        "10.196.59.198:17010",
        "10.196.59.199:17010",
        "10.196.59.200:17010"
     ],
     "disks": [
        "/data0:10737418240",
        "/data1:10737418240"
     ]
   }
   

For detailed explanations of datanode.json, please refer to Data Subsystem.

Start ObjectNode

nohup ./cfs-server -c objectnode.json &

Sample objectnode.json is shown as follows,

{
    "role": "objectnode",
    "domains": [
        "object.cfs.local"
    ],
    "listen": 17410,
    "masterAddr": [
       "10.196.59.198:17010",
       "10.196.59.199:17010",
       "10.196.59.200:17010"
    ],
    "logLevel": "info",
    "logDir": "/cfs/Logs/objectnode"
}

For detailed explanations of objectnode.json, please refer to Object Subsystem (ObjectNode).

Start Console

nohup ./cfs-server -c console.json &

Sample console.json is shown as follows,

{
    "role": "console",
    "logDir": "/cfs/log/",
    "logLevel": "debug",
    "listen": "80",
    "masterAddr": [
        "192.168.0.11:17010",
        "192.168.0.12:17010",
        "192.168.0.13:17010"
    ],
    "objectNodeDomain": "object.chubao.io",
    "monitor_addr": "http://192.168.0.102:9090",
    "dashboard_addr": "http://192.168.0.103",
    "monitor_app": "cfs",
    "monitor_cluster": "cfs"
}

For detailed explanations of console.json, please refer to Console.

Create Volume

By default, there are only a few data partitions allocated upon volume creation, and will be dynamically expanded according to actual usage.

curl -v "http://10.196.59.198:17010/admin/createVol?name=test&capacity=10000&owner=cfs"

For performance evaluation, extra data partitions shall be pre-created according to the amount of data nodes and disks to reach maximum performance.

curl -v "http://10.196.59.198:17010/dataPartition/create?name=test&count=120"

Mount Client

1. Run modprobe fuse to insert FUSE kernel module.

2. Run yum install -y fuse to install libfuse.

3. Run cfs-client -c fuse.json to start a client daemon.

   Sample fuse.json is shown as follows,

   {
     "mountPoint": "/cfs/mountpoint",
     "volName": "ltptest",
     "owner": "ltptest",
     "masterAddr": "10.196.59.198:17010,10.196.59.199:17010,10.196.59.200:17010",
     "logDir": "/cfs/client/log",
     "profPort": "17510",
     "exporterPort": "9504",
     "logLevel": "info"
   }
   

For detailed explanations of fuse.json, please refer to Client.

Note that end user can start more than one client on a single machine, as long as mountpoints are different.

Upgrading

1. freeze the cluster

curl -v "http://10.196.59.198:17010/cluster/freeze?enable=true"

2. upgrade each module
3. closed freeze flag

curl -v "http://10.196.59.198:17010/cluster/freeze?enable=false"

Resource Manager (Master)

The resource manager manages the file system by processing different types of tasks, such as creating/deleting/updating/loading partitions and keeping track of the resource status (such as the memory/disk utilization). The resource manager is also responsible for creating new volumes and adding new meta/data nodes to the ChubaoFS cluster. It has multiple replicas, among which the consistency is maintained by a consensus algorithm such as Raft, and persisted to a key value store such as RocksDB for backup and recovery.

Utilization-Based Distribution/Placement

The resource manager is a utilization-based distribution strategy to places the file metadata and contents across different meta and data nodes. Because each node can have multiple partitions, and the partitions in a volume do not need to reside on the same node, this distribution can be controlled at a finer granularity, resulting a more efficient resource management. Specifically, the distribution of file metadata and contents works follows:

First, when mounting a volume, the client asks the resource manager for a set of available meta and data partitions. These partitions are usually the ones on the nodes with the lowest memory/disk utilizations. Later on, when writing a file, the client can only choose the meta and data partitions in a random fashion from the ones allocated by the resource manager.

Second, when the resource manager finds that all the partitions in a volume is about to be full, it automatically adds a set of new partitions to this volume. These partitions are usually the ones on the nodes with the lowest memory/disk utilizations. Note that, when a partition is full, or a threshold (i.e., the number of files on a meta partition or the number of extents on a data partition) is reached, no new data can be stored on this partition, although it can still be modified or deleted.

Replica Placement

When choosing partitions for the replicas, the resource manager ensures that two replicas of the same partition never reside on the same node.

Meta Partition Splitting

There is a special requirement when splitting a meta partition. In particular, if a meta partition is about to reach its upper limit of the number of stored inodes and dentries, a splitting task needs to be performed with the requirement to ensure that the inode ids stored at the newly created partition are unique from the ones stored at the original partition.

To meet this requirement, when splitting a meta partition, the resource manager cuts off the inode range of the meta partition in advance at a upper bound end, a value greater than highest inode id used so far (denoted as maxInodeID), and sends a split request to the meta node to (1) update the inode range from 1 to end for the original meta partition, and (2) create a new meta partition with the inode range from end + 1 to infinity for this volume. As a result, the inode range for these two meta partitions becomes [1, end] and [end + 1, infinity], respectively. If there is another file needs to be created, then its inode id will be chosen as maxInodeID + 1 in the original meta partition, or end + 1 in the newly created meta partition. The maxInodeID of each meta partition can be obtained by the periodical communication between the resource manager and the the meta nodes.

Exception Handling

When a request to a meta/data partition times out (e.g., due to network outage), the remaining replicas of this partition are marked as read-only. When a meta/data partition is no longer available (e.g., due to hardware failures), all the data on this partition will eventually be migrated to a new available partition manually. This unavailability is identified by the multiple failures reported by the node when operating the files.

Metadata Subsystem

The metadata operations could make up as much as half of typical file system workloads. this can be important as there could be heavy accesses to the metadata of files by tens of thousands of clients simultaneously. A single node that stores the file metadata could easily become the performance/storage bottleneck. As a result, we employ a distributed metadata subsystem to manage the file metadata. In this way, the metadata requests from different clients can be forwarded to different nodes, which improves the scalability of the entire system.

Internal Structure

The metadata subsystem can be considered as an in-memory datastore of the file metadata. It can have thousands of meta nodes, each of which can have a set of meta partitions. Each meta partition on a meta node stores the file metadata in memory by maintaining a set of inodes and a set of dentries.

Generally speaking, An inode is an object that represents the underlying file (or directory), and can be identified by an unsigned 64-bit integer called the inode id. A dentry is an object that represents the directory hierarchy and can be identified by a string name and the id of the parent inode. For example, if we have two directories foo and bar, where foo is the parent directory of bar, then there are two inodes: one for foo called i1, and the other for bar called i2, and one dentry to represent the hierarchy of these two directories where i2 is the current inode and i1 is the parent inode.

A meta partition can only store the inodes and dentries of the files from the same volume. We employ two b-trees called inodeTree and dentryTree for fast lookup of inodes and dentries in the memory. The inodeTree is indexed by the inode id, and the dentryTree is indexed by the dentry name and the parent inode id. We also maintain a range of the inode ids (denoted as start and end) stored on a meta partition for splitting (see Resource Manager (Master)).

Replication

The replication during file write is performed in terms of meta partitions. The replication consistency is ensured by a revision of the Raft consensus protocol called the MultiRaft, which has the advantage of reduced heartbeat network traffic comparing to the original version.

Failure Recovery

The in-memory meta partitions are persisted to the local disk by snapshots and logs for backup and recovery. Some techniques such as log compaction are used to reduce the log files sizes and shorten the recovery time.

It is worth noting that, a failure that happens during a metadata operation could result an orphan inode with which has no dentry to be associated. The memory and disk space occupied by this inode can be hard to free. To minimize the chance of this case to happen, the client always issues a retry after a failure until the request succeeds or the maximum retry limit is reached.

Data Subsystem

The data subsystem is optimized for the storage of large and small files, which can be accessed in a sequential or random fashion.

Features

• Large File Storage

For large files, the contents are stored as a sequence of one or multiple extents, which can be distributed across different data partitions on different data nodes. Writing a new file to the extent store always causes the data to be written at the zero-offset of a new extent, which eliminates the need for the offset within the extent. The last extent of a file does not need to fill up its size limit by padding (i.e., the extent does not have holes), and never stores the data from other files.

• Small File Storage

The contents of multiple small files are aggregated and stored in a single extent, and the physical offset of each file content in the extent is recorded in the corresponding meta node. ChubaoFS relies on the punch hole interface, textit{fallocate()}footnote{url{http://man7.org/linux/man-pages/man2/fallocate.2.html}}, to textit{asynchronous} free the disk space occupied by the to-be-deleted file. The advantage of this design is to eliminate the need of implementing a garbage collection mechanism and therefore avoid to employ a mapping from logical offset to physical offset in an extent~cite{haystack}. Note that this is different from deleting large files, where the extents of the file can be removed directly from the disk.

• Replication

  The replication is performed in terms of partitions during file writes. Depending on the file write pattern, ChubaoFS adopts different replication strategies.

  When a file is sequentially written into ChubaoFS, a primary-backup replication protocol is used to ensure the strong consistency with optimized IO throughput.

  

  When overwriting an existing file portion during random writes, we employ a MultiRaft-based replication protocol, which is similar to the one used in the metadata subsystem, to ensure the strong consistency.

  

• Failure Recovery

  Because of the existence of two different replication protocols, when a failure on a replica is discovered, we first start the recovery process in the primary-backup-based replication by checking the length of each extent and making all extents aligned. Once this processed is finished, we then start the recovery process in our MultiRaft-based replication.

HTTP APIs

API	Method	Parameters	Description
/disks	GET	N/A	Get disk list and informations.
/partitions	GET	N/A	Get parttion list and infomartions.
/partition	GET	partitionId[int]	Get detail of specified partition.
/extent	GET	partitionId[int]&extentId[int]	Get extent informations.
/stats	GET	N/A	Get status of the datanode.

Object Subsystem (ObjectNode)

The object subsystem provides S3-compatible object storage interface. It makes possible for ChubaoFS becomes a fusion storage that expose two type interface (POSIX and S3-compatible). So that user can operate files stored in ChubaoFS by using native Amazon S3 SDKs.

Structure

The ObjectNode is a functional subsystem node. It fetch volume view (volume topology) on demand from Resource Manager (Master). Each ObjectNode communicate with metadata subsystem (MetaNode) and data subsystem (DataNode) directly.

ObjectNode is stateless design with high scalability and it have ability to operate all files stored in the ChubaoFS cluster which it belong to directly without any volume-mount operation.

Features

• Provides object storage interface compatible with native Amazon S3 SDKs.
• Fusion storage expose two type interface (POSIX and S3-compatible).
• Stateless and high scalability

Semantic Transform

Based on our POSIX-compatible design. Every file operate request comes from object storage interface need to be made semantic transform to POSIX.

POSIX	Object Storage
Cluster	Region
Volume	Bucket
Path	Key

Example:

Put object ‘example/a/b.txt’ will be create and write data to file ‘/a/b.txt’ in volume ‘example’.

User

Before using the object storage service, you need to create a user through the Master. While creating users, AccessKey and SecretKey will be generated for each user.

ChubaoFS uses the field Owner of the volume as the user ID. There are two ways to create users:

• When creating a volume through the API of the Master, if there is no user with the same name as the owner of the volume in the cluster, a user with the user ID of Owner will be automatically created.
• Create a user by calling the user management API of the Master.

Authentication

The signature validation algorithm in object storage interface is fully compatible with native Amazon S3 service. The authentication consisting of AccessKey and SecretKey generated by Resource Manager (Master) with user creation, which can be obtained through the Master API. The AccessKey is a 16-character string unique in the entire ChubaoFS cluster.

The user has all access permissions to the volume owned by him. Users can grant other users specified permissions to access volumes under their own names. The permissions are divided into the following three categories:

• Readonly or readwrite permission.
• Permission for a single operation, such as GetObject, PutObject, etc.
• Custom permission.

When a user uses the object storage service to execute a certain operation, ChubaoFS will identify whether the user has the corresponding permission.

Invisible Temporary Data

In order to make write operation in object storage interface atomically. Every write operation will create and write data to an invisible temporary. The volume operator in ObjectNode puts file data into temporary which only have ‘inode’ without ‘dentry’ in metadata. When all the file data stored successfully, the volume operator create or update ‘dentry’ in metadata makes it visible to users.

Object Mode Conflict (Important)

The POSIX and object storage are two different types of storage product, and the object storage is a Key-Value pair storage service. So that the object with key ‘a/b/c’ and the object with key ‘a/b’ are different object without any conflict.

But ChubaoFS is based on POSIX design. According to semantic transformation rule, the ‘b’ part in key ‘a/b/c’ transform to folder ‘b’ under the folder ‘a’ , and in key ‘a/b’ transform to file ‘b’ under the folder ‘a’.

The object key like this is conflict in ChubaoFS.

When the object being operated has conflicts with the existing object due to the above mode inconsistency, the ObjectNode will return a 409 status code to the client.

Supported S3 Features

• File object operations.
• Directory object operations.
• Multipart upload.
• Parallel download for high-level SDK APIs.
• Tagging for bucket and object.
• User-defined metadata for object.
• IP address and network segment black and white list for bucket ACL.
• Signature Algorithm V2 and V4.
• Cross-Origin Resource Sharing (CORS).

Unsupported S3 Features

• Version
• Restore deleted objects
• Locking objects
• Lifecycle configuration for bucket and object.
• Hosting Websites
• Encryption
• BitTorrent

Supported APIs

API	Reference
AbortMultipartUpload	https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html
CompleteMultipartUpload	https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html
CopyObject	https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
CreateBucket	https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html
CreateMultipartUpload	https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html
DeleteBucket	https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucket.html
DeleteBucketCors	https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketCors.html
DeleteBucketPolicy	https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketPolicy.html
DeleteBucketTagging	https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketTagging.html
DeleteObject	https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html
DeleteObjects	https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html
DeleteObjectTagging	https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html
GetBucketAcl	https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketAcl.html
GetBucketCors	https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketCors.html
GetBucketLocation	https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html
GetBucketPolicy	https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketPolicy.html
GetBucketTagging	https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketTagging.html
GetObject	https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
GetObjectAcl	https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAcl.html
GetObjectTagging	https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html
HeadBucket	https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html
HeadObject	https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html
ListBuckets	https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html
ListMultipartUploads	https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html
ListObjects	https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html
ListObjectsV2	https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html
ListParts	https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html
PutBucketAcl	https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketAcl.html
PutBucketCors	https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketCors.html
PutBucketPolicy	https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketPolicy.html
PutBucketTagging	https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketTagging.html
PutObject	https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html
PutObjectAcl	https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectAcl.html
PutObjectTagging	https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html
UploadPart	https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html

Supported SDKs

Object Node provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.

Name	Language	Link
AWS SDK for Java	Java	https://aws.amazon.com/sdk-for-java/
AWS SDK for JavaScript	JavaScript	https://aws.amazon.com/sdk-for-browser/
AWS SDK for JavaScript in Node.js	JavaScript	https://aws.amazon.com/sdk-for-node-js/
AWS SDK for Go	Go	https://docs.aws.amazon.com/sdk-for-go/
AWS SDK for PHP	PHP	https://aws.amazon.com/sdk-for-php/
AWS SDK for Ruby	Ruby	https://aws.amazon.com/sdk-for-ruby/
AWS SDK for .NET	.NET	https://aws.amazon.com/sdk-for-net/
AWS SDK for C++	C++	https://aws.amazon.com/sdk-for-cpp/
Boto3	Python	http://boto.cloudhackers.com

Client

The client runs on each container executing application code and exposes a file system interface to applications and can access a mounted volume via a user-space file system interface such as FUSE.

Client Caching

The client process runs entirely in the user space with its own cache, which has been used in the following cases.

To reduce the communication with the resource manager, the client caches the addresses of the available meta and data partitions assigned to the mounted volume, which can be obtained at the startup, and periodically synchronizes this available partitions with the resource manager.

To reduce the communication with the meta nodes, the client also caches the returned inodes and dentries when creating new files, as well as the data partition id, the extent id and the offset, after the file has been written to the data node successfully. When a file is opened for read/write, the client will force the cache metadata to be synchronous with the meta node.

To reduce the communication with the data nodes, the client caches the most recently identified leader. Our observation is that, when reading a file, the client may not know which data node is the current leader because the leader could change after a failure recovery. As a result, the client may try to send the read request to each replica one by one until a leader is identified. However, since the leader does not change frequently, by caching the last identified leader, the client can have minimized number of retries in most cases.

Integration with FUSE

The ChubaoFS client has been integrated with FUSE to provide a file system interface in the user space. In the past, low performance is considered the main disadvantage of such user-space file systems. But over the years, FUSE has made several improvement on its performance such as multithreading and write-back cache. In the future, we plan to develop our own POSIX-compliant file system interface in the kernel space to completely eliminate the overhead from FUSE.

Currently the write-back cache feature does not work well in ChubaoFS due to the following reason. The default write behavior of FUSE is called directIO, which bypasses the kernel’s page cache. This results in performance problems on writing small files as each write pushes the file data to the user daemon. The solution FUSE implemented was to make the page cache support a write-back policy that aggregates small data first, and then make writes asynchronous. With that change, file data can be pushed to the user daemon in larger blobs at a time. However, in real production, we found that the write-back cache is not very useful, because a write operation usually invoke another process that tries to balance the dirty pages (pages in the main memory that have been modified during writing data to disk are marked as “dirty” and have to be flushed to disk before they can be freed), which incurs extra overhead. This overhead becomes more obvious when small files are continuously written through FUSE.

Resource Manager (Master)

The cluster contains dataNodes,metaNodes,vols,dataPartitions and metaPartitions,they are managed by master server. The master server caches the metadata in mem,persist to GoLevelDB,and ensure consistence by raft protocol.

The master server manages dataPartition id to dataNode server mapping,metaPartition id to metaNode server mapping.

At lease 3 master nodes are required in respect to high availability.

Features

• Multi-tenant, Resource Isolation
• dataNodes,metaNodes shared,vol owns dataPartition and metaPartition exclusive
• Asynchronous communication with dataNode and metaNode

Configurations

ChubaoFS use JSON as configuration file format.

Properties :header: “Key”, “Type”, “Description”, “Mandatory”

role	string	Role of process and must be set to master	Yes
ip	string	host ip	Yes
listen	string	Http port which api service listen on	Yes
prof	string	golang pprof port	Yes
id	string	identy different master node	Yes
peers	string	the member information of raft group	Yes
logDir	string	Path for log file storage	Yes
logLevel	string	Level operation for logging. Default is error.	No
retainLogs	string	the number of raft logs will be retain.	Yes
walDir	string	Path for raft log file storage.	Yes
storeDir	string	Path for RocksDB file storage,path must be exist	Yes
clusterName	string	The cluster identifier	Yes
exporterPort	int	The prometheus exporter port	No
consulAddr	string	The consul register addr for prometheus exporter	No
metaNodeReservedMem	string	If the metanode memory is below this value, it will be marked as read-only. Unit: byte. 1073741824 by default.	No
heartbeatPort	string	Raft heartbeat port,5901 by default	No
replicaPort	string	Raft replica Port,5902 by default	No
nodeSetCap	string	the capacity of node set,18 by default	No
missingDataPartitionInterval	string	how much time it has not received the heartbeat of replica,the replica is considered missing ,24 hours by default	No
dataPartitionTimeOutSec	string	how much time it has not received the heartbeat of replica, the replica is considered not alive ,10 minutes by default	No
numberOfDataPartitionsToLoad	string	the maximum number of partitions to check at a time,40 by default	No
secondsToFreeDataPartitionAfterLoad	string	the task that release the memory occupied by loading data partition task can be start, only after secondsToFreeDataPartitionAfterLoad seconds ,300 by default	No
tickInterval	string	the interval of timer which check heartbeat and election timeout,500 ms by default	No
electionTick	string	how many times the tick timer has reset,the election is timeout,5 by default	No

Example:

{
 "role": "master",
 "id":"1",
 "ip": "10.196.59.198",
 "listen": "17010",
 "prof":"17020",
 "peers": "1:10.196.59.198:17010,2:10.196.59.199:17010,3:10.196.59.200:17010",
 "retainLogs":"20000",
 "logDir": "/cfs/master/log",
 "logLevel":"info",
 "walDir":"/cfs/master/data/wal",
 "storeDir":"/cfs/master/data/store",
 "exporterPort": 9500,
 "consulAddr": "http://consul.prometheus-cfs.local",
 "clusterName":"chubaofs01",
 "metaNodeReservedMem": "1073741824"
}

Start Service

nohup ./master -c config.json > nohup.out &

Meta Subsystem

Metanode is the manager of meta partitions and replicated by MultiRaft. Each metanode manages various of partitions. Each partition covers an inode range, and maintains two in-memory btrees: inode btree and dentry btree.

At lease 3 meta nodes are required in respect to high availability.

Properties

Key	Type	Description	Mandatory
role	string	Role of process and must be set to metanode	Yes
listen	string	Listen and accept port of the server	Yes
prof	string	Pprof port	Yes
localIP	string	IP of network to be choose	No. If not specified, the ip address used to communicate with the master is used.
logLevel	string	Level operation for logging. Default is error	No
metadataDir	string	MetaNode store snapshot directory	Yes
logDir	string	Log directory	Yes
raftDir	string	Raft wal directory	Yes
raftHeartbeatPort	string	Raft heartbeat port	Yes
raftReplicaPort	string	Raft replicate port	Yes
consulAddr	string	Addresses of monitor system	No
exporterPort	string	Port for monitor system	No
masterAddr	string	Addresses of master server	Yes
zoneName	string	Specified zone. default by default.	No
totalMem	string	Max memory metadata used. The value needs to be higher than the value of metaNodeReservedMem in the master configuration. Unit: byte	Yes
deleteBatchCount	int64	when deleting inodes, how many are deleted at a time ,500 by default	No

Example:

{
     "role": "metanode",
     "listen": "17210",
     "prof": "17220",
     "logLevel": "debug",
     "metadataDir": "/cfs/metanode/data/meta",
     "logDir": "/cfs/metanode/log",
     "raftDir": "/cfs/metanode/data/raft",
     "raftHeartbeatPort": "17230",
     "raftReplicaPort": "17240",
     "consulAddr": "http://consul.prometheus-cfs.local",
     "exporterPort": 9501,
     "totalMem":  "8589934592",
     "masterAddr": [
         "10.196.59.198:17010",
         "10.196.59.199:17010",
         "10.196.59.200:17010"
     ]
 }

Notice

• listen, raftHeartbeatPort, raftReplicaPort can’t be modified after boot startup first time;
• Above config would be stored under directory raftDir in constcfg file. If need modified forcely，you must delete this file manually;
• These configuration items associated with master’s metanode infomation . If they have been modified, master would’t be found old metanode;

Data Subsystem

How To Start DataNode

Start a DataNode process by execute the server binary of ChubaoFS you built with -c argument and specify configuration file. At least 4 data nodes are required in respect to high availability.

nohup cfs-server -c datanode.json &

Configurations

Properties

Key	Type	Description	Mandatory
role	string	Role of process and must be set to datanode	Yes
listen	string	Port of TCP network to be listen	Yes
localIP	string	IP of network to be choose	No,If not specified, the ip address used to communicate with the master is used.
prof	string	Port of HTTP based prof and api service	Yes
logDir	string	Path for log file storage	Yes
logLevel	string	Level operation for logging. Default is error	No
raftHeartbeat	string	Port of raft heartbeat TCP network to be listen	Yes
raftReplica	string	Port of raft replicate TCP network to be listen	Yes
raftDir	string	Path for raft log file storage	No
consulAddr	string	Addresses of monitor system	No
exporterPort	string	Port for monitor system	No
masterAddr	string slice	Addresses of master server	Yes
zoneName	string	Specified zone. default by default.	No
disks	string slice	Format: PATH:RETAIN. PATH: Disk mount point. RETAIN: Retain space. (Ranges: 20G-50G.)	Yes

Example:

{
    "role": "datanode",
    "listen": "17310",
    "prof": "17320",
    "logDir": "/cfs/datanode/log",
    "logLevel": "info",
    "raftHeartbeat": "17330",
    "raftReplica": "17340",
    "raftDir": "/cfs/datanode/log",
    "consulAddr": "http://consul.prometheus-cfs.local",
    "exporterPort": 9502,
    "masterAddr": [
         "10.196.59.198:17010",
         "10.196.59.199:17010",
         "10.196.59.200:17010"
    ],
     "disks": [
        "/data0:10737418240",
        "/data1:10737418240"
    ]
}

Notice

• Since datanode uses SEEK_HOLE and SEEK_DATA operations which is supported by XFS (since Linux 3.5) and ext4 (since Linux 3.8), users should pay attention to the Linux kernel version on which datanodes are deployed.
• listen, raftHeartbeat, raftReplica can’t be modified after boot startup first time.
• Above config would be stored under directory raftDir in constcfg file. If need modified forcely, you must delete this file manually.
• These configuration items associated with master’s datanode infomation. If they have been modified, master would’t be found old datanode.

Object Subsystem (ObjectNode)

How To start ObjectNode

Start a ObjectNode process by execute the server binary of ChubaoFS you built with -c argument and specify configuration file.

nohup cfs-server -c objectnode.json &

Note: If you do not intend to use the object storage service, you do not need to start the ObjectNode.

Configurations

Object Node using JSON format configuration file.

Properties

Key	Type	Description	Mandatory
role	string	Role of process and must be set to objectnode	Yes
listen	string	Listen and accept port of the server	Yes
domains	string slice	Domain of S3-like interface which makes wildcard domain support Format: DOMAIN	No
logDir	string	Log directory	Yes
logLevel	string	Level operation for logging. Default: error	No
masterAddr	string slice	Format: HOST:PORT. HOST: Hostname, domain or IP address of master (resource manager). PORT: port number which listened by this master	Yes
authNodes	string slice	Format: HOST:PORT. HOST: Hostname, domain or IP address of AuthNode. PORT: port number which listened by this AuthNode	Yes
exporterPort	string	Port for monitor system	No
prof	string	Pprof port	Yes

Example:

{
     "role": "objectnode",
     "listen": "17410",
     "domains": [
         "object.cfs.local"
     ],
     "logDir": "/cfs/Logs/objectnode",
     "logLevel": "info",
     "masterAddr": [
         "10.196.59.198:17010",
         "10.196.59.199:17010",
         "10.196.59.200:17010"

     ],
     "exporterPort": 9503,
     "prof": "7013"
}

Fetch Authentication Keys

First, you need create a user through User API, and then get keys information. Refer to User.

You can also use Command Line Interface (CLI) tool to create and get user’s AccessKey and SecretKey:

$ cli user create [USER_ID]
$ cli user info [USER_ID]

Using Object Storage Interface

Object Subsystem (ObjectNode) provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.

For detail about list of supported APIs, see Supported S3-compatible APIs at Object Subsystem (ObjectNode)

For detail about list of supported SDKs, see Supported SDKs at Object Subsystem (ObjectNode)

Using S3cmd

Use s3cmd to access the ObjectNode deployed locally.

Installation

Install the s3cmd from https://s3tools.org/s3cmd .

Configuration

Edit s3cmd configuration file $HOME/.s3cfg

host_base = 127.0.0.1
host_bucket = 127.0.0.1
use_https = False
access_key = YOUR_ACCESS_KEY
secret_key = YOUR_SECRET_KEY

Example: making a bucket (volume)

s3cmd mb s3://my_volume
Bucket 's3://my_volume/' created

Example: uploading an local file to ChubaoFS

s3cmd put data.dat s3://my_volume
upload: 'data.dat' -> 's3://my_volume/data.dat'

Example: listing buckets (volumes)

s3cmd ls
2020-05-10 15:29 s3://my_volume

Example: listing files stored in a ChubaoFS volume

s3cmd ls s3://my_volume
                         DIR    s3://my_volume/backup/
2020-05-10  15:31   10485760    s3://my_volume/data.dat
2020005-10  15:33         10    s3://my_volume/hello.txt

Example: deleting file stored in a ChubaoFS volume

s3cmd rm s3://my_volume/data.dat
delete: 's3://my_volume/data.dat'

Detail usage for s3cmd see https://s3tools.org/usage .

Using AWS Java SDK

Use AWS Java SDK to access the ObjectNode deployed locally.

Install by Maven:

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>s3</artifactId>
    <version>2.10.71</version>
</dependency>

Example: uploading file to ChubaoFS volume (PutObject)

Regions clientRegion = Region.of("*** Region name ***"); // Setup region (the cluster name)
String endpoint = "http://127.0.0.1"; // Setup endpoint
String accessKey = "*** Access Key ***"; // Setup AccessKey
String secretKey = "*** Secret Key ***"; // Setup SecretKey
String bucketName = "*** Bucket name ***"; // Setup target bucket (ChubaoFS Volume)
String objectKey = "*** File object key name ***"; // Setup object key
[]byte data = "*** Example File Data as String **".getBytes();

try {
    // Setup credential
    AwsCredentialsProvider credentialsProvider = StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey));

    // Init S3 client
    S3Configuration configuration = S3Configuration.builder()
        .chunkedEncodingEnabled(true)
        .build();
    S3Client client = S3Client.builder()
        .region(region)
        .credentialsProvider(credentialsProvider)
        .endpointOverride(URI.create(endpoint))
        .serviceConfiguration(configuration)
        .build();

    // Upload file
    PutObjectRequest request = PutObjectRequest.builder()
        .bucket(bucketName)
        .key(objectKey)
        .build();
    RequestBody body = RequestBody.fromBytes(data);
    s3Client.putObject(request, body)
} catch (Exception e) {
    e.printStackTrace();
}

Example: downloading file stored in ChubaoFS volume (GetObject)

Regions clientRegion = Region.of("*** Region name ***"); // Setup region
String endpoint = "http://127.0.0.1"; // Setup endpoint
String accessKey = "*** Access Key ***"; // Setup AccessKey
String secretKey = "*** Secret Key ***"; // Setup SecretKey
String bucketName = "*** Bucket name ***"; // Setup target bucket (ChubaoFS Volume)
String objectKey = "*** File object key name ***"; // Setup object key

try {
    // Setup credential
    AwsCredentialsProvider credentialsProvider = StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey));

    // Init S3 client
    S3Configuration configuration = S3Configuration.builder()
        .chunkedEncodingEnabled(true)
        .build();
    S3Client client = S3Client.builder()
        .region(region)
        .credentialsProvider(credentialsProvider)
        .endpointOverride(URI.create(endpoint))
        .serviceConfiguration(configuration)
        .build();

    // Get file data
    GetObjectRequest request = GetObjectRequest.builder()
        .bucket(bucketName)
        .key(objectKey)
        .build();

    InputStream is = s3Client.getObject(request)
    while (true) {
        if (is.read() == -1) {
            break
        }
    }
} catch (Exception e) {
    e.printStackTrace();
}

Using AWS Python SDK (Boto3)

Use AWS Python SDK (Boto3) to access the ObjectNode deployed locally.

Install Boto3 by PIP:

$ pip install boto3

Example: uploading file to ChubaoFS volume (PutObject)

import boto3

endpoint = " ** endpoint url ** "  # example: http://127.0.0.1
region_name = " ** region name ** "
access_key = " ** your access key ** "  # your access key
secret_key = " ** your secret key ** "  # your secret key
bucket = " ** your bucket (volume) ** "  # your volume name
key = " ** your object key (file path in CFS) ** "  # your object name

def put_file():
    session = boto3.Session(
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key)
    client = session.client(service_name="s3", region_name=region_name, endpoint_url=endpoint)
    client.put_object(Bucket=bucket, Key=key, Body=bytes("hello world"))

Example: downloading file stored in ChubaoFS volume (GetObject)

import boto3

endpoint = " ** endpoint url ** "  # example: http://127.0.0.1
region_name = " ** region name ** "
access_key = " ** your access key ** "  # your access key
secret_key = " ** your secret key ** "  # your secret key
bucket = " ** your bucket (volume) ** "  # your volume name
key = " ** your object key (file path in CFS) ** "  # your object name

def get_file():
    session = boto3.Session(
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key)
    client = session.client(service_name="s3", region_name=region_name, endpoint_url=endpoint)
    result = client.get_object(Bucket=bucket, Key=key)
    print(result["Body"].read())

Console

How To Start Console

Start a Console process by execute the server binary of ChubaoFS you built with -c argument and specify configuration file.

nohup cfs-server -c console.json &

Configurations

Properties

Key	Type	Description	Mandatory
role	string	Role of process and must be set to console	Yes
logDir	string	Path for log file storage	Yes
logLevel	string	Level operation for logging. Default is error	No
listen	string	Port of TCP network to be listen, default is 80	Yes
masterAddr	string slice	Addresses of master server	Yes
objectNodeDomain	string	object domain for sign url for down	Yes
monitor_addr	string	Prometheus the address	Yes
dashboard_addr	string	console menu forward to Grafana	Yes
monitor_app	string	the tag for monitor, it same as master config	Yes
monitor_cluster	string	the tag for monitor, it same as master config	Yes

Example:

{
  "role": "console",
  "logDir": "/cfs/log/",
  "logLevel": "debug",
  "listen": "80",
  "masterAddr": [
    "192.168.0.11:17010",
    "192.168.0.12:17010",
    "192.168.0.13:17010"
  ],
  "monitor_addr": "http://192.168.0.102:9090",
  "dashboard_addr": "http://grafana.chubao.io",
  "monitor_app": "cfs",
  "monitor_cluster": "spark"
}

Notice

• add 127.0.0.1 console.chubao.io grafana.chubao.io in /etc/hosts
• you can visit it by http://127.0.0.1:80
• in console default user is root default password is ChubaoFSRoot
• If you are upgrading your ChubaoFS from v2.0.0 or earlier version, the password may not be compatible, you can use curl -H “Content-Type:application/json” -X POST –data ‘{“id”:”testuser”,”pwd”:”12345”,”type”:2}’ “http://10.196.59.198:17010/user/create” to create new user to use it

Client

Prerequisite

Insert FUSE kernel module and install libfuse.

modprobe fuse
yum install -y fuse

Prepare Config File

fuse.json

{
  "mountPoint": "/cfs/mountpoint",
  "volName": "ltptest",
  "owner": "ltptest",
  "masterAddr": "10.196.59.198:17010,10.196.59.199:17010,10.196.59.200:17010",
  "logDir": "/cfs/client/log",
  "logLevel": "info",
  "profPort": "27510"
}

Supported Configurations

Name	Type	Description	Mandatory
mountPoint	string	Mount point	Yes
volName	string	Volume name	Yes
owner	string	Owner name as authentication	Yes
masterAddr	string	Resource manager IP address	Yes
logDir	string	Path to store log files	No
logLevel	string	Log level：debug, info, warn, error	No
profPort	string	Golang pprof port	No
exporterPort	string	Performance monitor port	No
consulAddr	string	Performance monitor server address	No
lookupValid	string	Lookup valid duration in FUSE kernel module, unit: sec	No
attrValid	string	Attr valid duration in FUSE kernel module, unit: sec	No
icacheTimeout	string	Inode cache valid duration in client	No
enSyncWrite	string	Enable DirectIO sync write, i.e. make sure data is fsynced in data node	No
autoInvalData	string	Use AutoInvalData FUSE mount option	No
rdonly	bool	Mount as read-only file system	No
writecache	bool	Leverage the write cache feature of kernel FUSE. Requires the kernel FUSE module to support write cache.	No
keepcache	bool	Keep kernel page cache. Requires the writecache option is enabled.	No
token	string	Specify the capability of a client instance.	No
readRate	int	Read Rate Limit. Unlimited by default.	No
writeRate	int	Write Rate Limit. Unlimited by default.	No
followerRead	bool	Enable read from follower. False by default.	No
accessKey	string	Access key of user who owns the volume.	No
secretKey	string	Secret key of user who owns the volume.	No
disableDcache	bool	Disable Dentry Cache. False by default.	No
subdir	string	Mount sub directory.	No
fsyncOnClose	bool	Perform fsync upon file close. True by default.	No
maxcpus	int	The maximum number of available CPU cores. Limit the CPU usage of the client process.	No
enableXattr	bool	Enable xattr support. False by default.	No

Mount

Use the example fuse.json, and client is mounted on the directory /mnt/fuse. All operations to /mnt/fuse would be performed on the backing distributed file system.

./cfs-client -c fuse.json

Unmount

It is recommended to use standard Linux umount command to terminate the mount.

Monitor

ChubaoFS use prometheus as metrics collector. It simply config as follow in master, metanode, datanode, client’s config file：

{
    "exporterPort": 9505,
    "consulAddr": "http://consul.prometheus-cfs.local"
}

• exporterPort：prometheus exporter Port. when set, can export prometheus metrics from URL(http://$hostip:$exporterPort/metrics). If not set, prometheus exporter will unavailable；
• consulAddr: consul register address, it can work with prometheus to auto discover deployed ChubaoFS nodes, if not set, consul register will not work.

Using grafana as prometheus metrics web front：

Also, user can use prometheus alertmanager to capture and route alerts to the correct receiver. please visit prometheus alertmanger web-doc

Metrics

• Cluster

  • The number of nodes: MasterCount , MetaNodeCount , DataNodeCount , ObjectNodeCount
  • The number of clients: ClientCount
  • The number of volumes: VolumeCount
  • The size of nodes: DataNodeSize , MetaNodeSize
  • The used ratio of nodes: DataNodeUsedRatio , MetaNodeUsedRatio
  • The number of inactive nodes: DataNodeInactive , MetaNodesInactive
  • The capacity of volumes: VolumeTotalSize
  • The used ratio of volumes: VolumeUsedRatio
  • The number of error disks: DiskError

• Volume

  • The used size of volumes: VolumeUsedSize
  • The used ratio of volumes: VolumeUsedRatio
  • The capacity change rate of volumes: VolumeSizeRate

• Master

  • The number of invalid masters: master_nodes_invalid
  • The number of invalid metanodes: metanode_inactive
  • The number of invalid datanodes:： datanode_inactive
  • The number of invalid clients:： fuseclient_inactive

• MetaNode

  • The duration of each operation (Time) and the number of operations per second (Ops) on the metanode, which can be selected from the MetaNodeOp drop-down list.

• DataNode

  • The duration of each operation (Time) and the number of operations per second (Ops) on the datanode, which can be selected from the DataNodeOp drop-down list.

• ObjectNode

  • The duration of each operation (Time) and the number of operations per second (Ops) on the objectnode, which can be selected from the objectNodeOp drop-down list.

• FuseClient

  • The duration of each operation (Time) and the number of operations per second (Ops) on the client, which can be selected from the fuseOp drop-down list.

Recommended focus metrics: cluster status, node or disk failure, total size, growth rate, etc.

Grafana DashBoard Config

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "gnetId": null,
  "graphTooltip": 0,
  "id": 21,
  "iteration": 1578879879321,
  "links": [],
  "panels": [
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 40,
      "panels": [],
      "title": "Cluster",
      "type": "row"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": null,
      "description": "master node total count of cluster",
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 0,
        "y": 1
      },
      "id": 38,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"master\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "MasterCount",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": null,
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 4,
        "y": 1
      },
      "id": 42,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"metanode\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "MetanodeCount",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": "default",
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 8,
        "y": 1
      },
      "id": 41,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "count(go_info{cluster=\"$cluster\", app=\"$app\", role=\"dataNode\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "DatanodeCount",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": "default",
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 3,
        "x": 12,
        "y": 1
      },
      "id": 86,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"fuseclient\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "ClientCount",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": "default",
      "description": "Cluster volume total count",
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 3,
        "x": 15,
        "y": 1
      },
      "id": 93,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "sum([[app]]_master_vol_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "VolumeCount",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": null,
      "decimals": null,
      "format": "dateTimeFromNow",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 3,
        "x": 18,
        "y": 1
      },
      "id": 113,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "process_start_time_seconds{cluster=\"$cluster\", app=\"$app\", instance=~\"$instance\"}*1000",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "ProcessStartTime",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 5
      },
      "id": 115,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum(go_info{app=\"$app\", cluster=\"$cluster\"}) by (cluster, app, role)",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{cluster}}-{{role}}",
          "refId": "A"
        },
        {
          "expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"$role\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "[[role]]-count",
          "refId": "B"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "NodeAlive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 7,
        "y": 5
      },
      "id": 179,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "sum"
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum(cfs_master_disk_error{app=\"$app\", cluster=\"$cluster\"} > 0)",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "sum",
          "refId": "D"
        },
        {
          "expr": "cfs_master_disk_error{app=\"$app\", cluster=\"$cluster\"} > 0",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{addr}}_{{path}}",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "DiskError",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 14,
        "y": 5
      },
      "id": 180,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "cfs_master_metaNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{instance}}",
          "refId": "B"
        },
        {
          "expr": "sum(cfs_master_metaNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "sum",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetaNodesInactive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 11
      },
      "id": 181,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "cfs_master_dataNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{instance}}",
          "refId": "C"
        },
        {
          "expr": "sum(cfs_master_dataNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "sum",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "DatanodeInactive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 7,
        "y": 11
      },
      "id": 177,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/used_ratio.*/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "cfs_master_dataNodes_total_GB{app=\"$app\",cluster=\"$cluster\"}>0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "TotalGB",
          "refId": "B"
        },
        {
          "expr": "cfs_master_dataNodes_used_GB{app=\"$app\",cluster=\"$cluster\"}>0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "UsedGB",
          "refId": "A"
        },
        {
          "expr": "cfs_master_dataNodes_increased_GB{app=\"$app\",cluster=\"$cluster\"}>0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "IncreasedGB",
          "refId": "C"
        },
        {
          "expr": "sum(cfs_master_dataNodes_used_GB{app=\"$app\",cluster=\"$cluster\"}) / sum(cfs_master_dataNodes_total_GB{app=\"$app\",cluster=\"$cluster\"})",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "UsedRatio",
          "refId": "D"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "DatanodeSize",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "decgbytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "percentunit",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 14,
        "y": 11
      },
      "id": 92,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_master_metaNodes_total_GB{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "total_GB",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_master_metaNodes_used_GB{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "used_GB",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_master_metaNodes_increased_GB{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "increased_GB",
          "refId": "C"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetanodeSize",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "decgbytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 17
      },
      "id": 175,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_ratio.*/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "cfs_master_vol_usage_ratio{app=\"$app\",cluster=\"$cluster\"}",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "{{volName}}_usage_ratio",
          "refId": "B"
        },
        {
          "expr": "cfs_master_vol_total_GB{app=\"$app\",cluster=\"$cluster\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{volName}}_total_GB",
          "refId": "A"
        },
        {
          "expr": "cfs_master_vol_used_GB{app=\"$app\",cluster=\"$cluster\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{volName}}_used_GB",
          "refId": "C"
        },
        {
          "expr": "cfs_master_vol_used_GB{app=\"$app\",cluster=\"$cluster\"} / cfs_master_vol_total_GB{app=\"$app\",cluster=\"$cluster\"} ",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{volName}}_used_ratio",
          "refId": "D"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "VolumeSize",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "decgbytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "percentunit",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 7,
        "y": 17
      },
      "id": 90,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_master_vol_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "vol_count",
          "refId": "M"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "VolumeCount",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 14,
        "y": 17
      },
      "id": 73,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_QPS/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_metanode_OpMetaOpen{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Open",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaLookup{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Lookup",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaNodeHeartbeat{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "NodeHeartbeat",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaReadDir{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "ReadDir",
          "refId": "D"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaReleaseOpen{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "ReleaseOpen",
          "refId": "E"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaSetattr{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Setattr",
          "refId": "F"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Truncate",
          "refId": "G"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaOpen{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Open_QPS",
          "refId": "H"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaLookup{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Lookup_QPS",
          "refId": "I"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaReadDir{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "ReadDir_QPS",
          "refId": "K"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Truncate_QPS",
          "refId": "L"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaReleaseOpen{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "ReleaseOpen_QPS",
          "refId": "J"
        },
        {
          "expr": "sum(rate([[app]]cfs_metanode_OpMetaSetattr{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Setattr_QPS",
          "refId": "M"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetanodeOperations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 23
      },
      "id": 71,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_QPS/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_metanode_OpMetaBatchInodeGet{cluster=~\"$cluster\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "BatchGet",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaCreateInode{cluster=~\"$cluster\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Create",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaDeleteInode{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Delete",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaEvictInode{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Evict",
          "refId": "D"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaInodeGet{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "GetInode",
          "refId": "E"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaLinkInode{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "LinkInode",
          "refId": "F"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaCreateInode{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateInode_QPS",
          "refId": "G"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaDeleteInode{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "DeleteInode_QPS",
          "refId": "H"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaEvictInode{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "EvictInode_QPS",
          "refId": "I"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaInodeGet{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "GetInode_QPS",
          "refId": "J"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaLinkInode{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "LinkInode_QPS",
          "refId": "K"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpBatchInodeGet{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "BantchInodeGet_QPS",
          "refId": "L"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "TruncateInode",
          "refId": "M"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "TruncateInode_QPS",
          "refId": "N"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaUnlinkInode{cluster=~\"$cluster\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "UnlinkInode",
          "refId": "O"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaUnlinkInode{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "UnlinkInnode_QPS",
          "refId": "P"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetanodeInodeOperations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 7,
        "y": 23
      },
      "id": 45,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_QPS/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_metanode_OpMetaCreateDentry{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateDentryCreateDentry",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaDeleteDentry{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "DeleteDentry",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaUpdateDentry{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "UpdateDentry",
          "refId": "C"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaCreateDentry{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateDentry_QPS",
          "refId": "D"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaDeleteDentry{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "DeleteDentry_QPS",
          "refId": "E"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaUpdateDentry{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "UpdateDentry_QPS",
          "refId": "F"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetanodeDentryOperation",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 14,
        "y": 23
      },
      "id": 79,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_QPS/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_dataNode_OpCreateExtent{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateExtent",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_dataNode_OpMarkDelete{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "MarkDeleteExtent",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_dataNode_OpRandomWrite{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "RandomWrite",
          "refId": "D"
        },
        {
          "expr": "sum([[app]]_dataNode_OpWrite{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "AppendWrite",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_dataNode_OpStreamRead{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Read",
          "refId": "F"
        },
        {
          "expr": "sum(rate([[app]]_datanode_OpCreateExtent{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateExtent_QPS",
          "refId": "E"
        },
        {
          "expr": "sum(rate([[app]]_dataNode_OpMarkDelete{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "MarkDeleteExtent_QPS",
          "refId": "G"
        },
        {
          "expr": "sum(rate([[app]]_dataNode_OpRandomWrite{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "RandomWrite_QPS",
          "refId": "H"
        },
        {
          "expr": "sum(rate([[app]]_dataNode_OpWrite{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "AppendWrite_QPS",
          "refId": "I"
        },
        {
          "expr": "sum(rate([[app]]_dataNode_OpStreamRead{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Read_QPS",
          "refId": "J"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "DatanodeExtentOperations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 29
      },
      "id": 70,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_metanode_OpCreateMetaPartition{cluster=~\"$cluster\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Create",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_metanode_OpLoadMetaPartition{cluster=~\"$cluster\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Load",
          "refId": "B"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetapartitionOperations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 7,
        "y": 29
      },
      "id": 75,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_dataNode_OpLoadDataPartition{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "LoadDataPartition",
          "refId": "G"
        },
        {
          "expr": "sum([[app]]_dataNode_OpDataNodeHeartbeat{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "DataNodeHeartbeat",
          "refId": "F"
        },
        {
          "expr": "sum([[app]]_dataNode_OpCreateDataPartition{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateDataPartition",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_dataNode_OpAddDataPartitionRaftMember{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "AddDataPartitionRaftMember",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_dataNode_OpDeleteDataPartition{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "DeleteDataPartition",
          "refId": "C"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "DatapartitionOperations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 35
      },
      "id": 34,
      "panels": [],
      "title": "Master",
      "type": "row"
    },
    {
      "aliasColors": {},
      "bars": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 7,
        "w": 7,
        "x": 0,
        "y": 36
      },
      "id": 162,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": false,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "master_count",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "go_info{cluster=\"$cluster\", role=\"master\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "[[cluster]]_master_{{instance}}",
          "refId": "C"
        },
        {
          "expr": "count(go_info{cluster=~\"$cluster\", role=~\"master\"})",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "[[cluster]]_master_count",
          "refId": "G"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "[[role]]_nodes",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 7,
        "w": 7,
        "x": 7,
        "y": 36
      },
      "id": 163,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": false,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "expr": "go_info{cluster=\"$cluster\", role=\"metanode\"}",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "meta_{{instance}}",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_master_metaNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "metaNode",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_master_dataNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode",
          "refId": "A"
        },
        {
          "expr": "[[app]]_master_metaNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "meta_count",
          "refId": "D"
        },
        {
          "expr": "[[app]]_master_dataNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode_count",
          "refId": "E"
        },
        {
          "expr": "count(process_start_time_seconds{role=\"fuseclient\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "client_count",
          "refId": "F"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "[[cluster]]_metanode_alive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 7,
        "w": 7,
        "x": 0,
        "y": 43
      },
      "id": 164,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": false,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "expr": "go_info{cluster=\"$cluster\", role=\"dataNode\"}",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{instance}}",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_master_metaNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "metaNode",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_master_dataNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode",
          "refId": "A"
        },
        {
          "expr": "[[app]]_master_metaNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "meta_count",
          "refId": "D"
        },
        {
          "expr": "[[app]]_master_dataNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode_count",
          "refId": "E"
        },
        {
          "expr": "count(process_start_time_seconds{role=\"fuseclient\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "client_count",
          "refId": "F"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "[[cluster]]_datanode_alive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 7,
        "w": 7,
        "x": 7,
        "y": 43
      },
      "id": 165,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": false,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "expr": "go_info{cluster=\"$cluster\", role=\"fuseclient\"}",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "fuseclient_{{instance}}",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_master_metaNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "metaNode",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_master_dataNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode",
          "refId": "A"
        },
        {
          "expr": "[[app]]_master_metaNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "meta_count",
          "refId": "D"
        },
        {
          "expr": "[[app]]_master_dataNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode_count",
          "refId": "E"
        },
        {
          "expr": "count(process_start_time_seconds{role=\"fuseclient\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "client_count",
          "refId": "F"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "[[cluster]]_fuseclient_alive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "collapsed": true,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 50
      },
      "id": 36,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 3
          },
          "id": 117,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_metanode_OpMetaCreateInode_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "InodeCreate",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 3
          },
          "id": 44,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteInode_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaEvictInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaInodeGet_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLinkInode_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "InodeBatchGet",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 3
          },
          "id": 119,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteInode_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaEvictInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaInodeGet_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLinkInode_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_metanode_OpMetaLinkInode_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLinkInode_ops_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "InodeLink",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 11
          },
          "id": 120,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteInode_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaEvictInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaInodeGet_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLinkInode_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "InodeEvict",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "description": "",
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 11
          },
          "id": 121,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpCreateMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpCreateMetaPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpLoadMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpLoadMetaPartition_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "LoadMetaPartition",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 11
          },
          "id": 58,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpCreateMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpCreateMetaPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpLoadMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpLoadMetaPartition_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "CreateMetaPartition",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 19
          },
          "id": 88,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaExtentsList{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsList_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaExtentsList",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 19
          },
          "id": 50,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaExtentsAdd{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsAdd_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaExtentsList{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsList_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaExtentsAdd",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 19
          },
          "id": 126,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaNodeHeartbeat",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 27
          },
          "id": 72,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaCreateDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaUpdateDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaCreateDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 27
          },
          "id": 125,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaOpen_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLookup_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReleaseOpen_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaSetattr_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaTruncate_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaLookup",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 27
          },
          "id": 122,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaCreateDentry{app=\"$app\",cluster=~\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteDentry{app=\"$app\",cluster=~\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaUpdateDentry{app=\"$app\",cluster=~\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaDeleteDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 35
          },
          "id": 124,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_metanode_OpMetaReadDir_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaReadDir",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 35
          },
          "id": 129,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaOpen_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLookup_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReleaseOpen_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaSetattr_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaTruncate_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaReleaseOpen",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 35
          },
          "id": 116,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteInode_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaEvictInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaInodeGet_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLinkInode_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaDeleteInode",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 43
          },
          "id": 127,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaOpen_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLookup_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReleaseOpen_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaSetattr_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaTruncate_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaTruncate",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 43
          },
          "id": 128,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaOpen_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLookup_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReleaseOpen_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaSetattr_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaTruncate_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaSetattr",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 43
          },
          "id": 123,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaCreateDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaUpdateDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaUpdateDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 51
          },
          "id": 46,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaOpen_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLookup_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReleaseOpen_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaSetattr_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaTruncate_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaOpen",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        }
      ],
      "title": "Metanode",
      "type": "row"
    },
    {
      "collapsed": true,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 51
      },
      "id": 27,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "description": "",
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 52
          },
          "id": 132,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_ops_{{instance}}",
              "refId": "Q"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "DatanodeHeartbeat",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 52
          },
          "id": 133,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpCreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "CreateDataPartition",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 52
          },
          "id": 137,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_dataNode_OpLoadDataPartition_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "LoadDataPartition",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 60
          },
          "id": 138,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "NotifyReplicasToRepair",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 60
          },
          "id": 134,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_dataNode_OpGetAllWatermarks_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "GetAllWatermarks",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 60
          },
          "id": 135,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_dataNode_OpGetAppliedId_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_ops_{{instance}}",
              "refId": "Q"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "GetAppliedId",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 68
          },
          "id": 142,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "rate([[app]]dataNode_OpStreamRead_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "StreamRead",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 68
          },
          "id": 143,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "Write",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 68
          },
          "id": 139,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MarkDelete",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 76
          },
          "id": 144,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "rate([[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "CreateExtent",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 76
          },
          "id": 145,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            },
            {
              "expr": "rate([[app]]_dataNode_OpExtentRepairRead_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "ExtentRepairdRead",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 76
          },
          "id": 74,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_ops_{{instance}}",
              "refId": "Q"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "BroadcastMinAppliedID",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 84
          },
          "id": 141,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "rate([[app]]_dataNode_OpReadTinyDelete_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "ReadTinyDelete",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 84
          },
          "id": 136,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "GetPartitionSize",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 84
          },
          "id": 146,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpTinyExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "TinyExtentRepairdRead_{{instance}}",
              "refId": "P"
            },
            {
              "expr": "rate([[app]]_dataNode_OpTinyExtentRepairRead_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "TinyExtentRepairdRead_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "TinyExtentRepairdRead",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 92
          },
          "id": 131,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "DecommissionDataPartition",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 92
          },
          "id": 140,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "RandomWrite",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        }
      ],
      "title": "Datanode",
      "type": "row"
    },
    {
      "collapsed": true,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 52
      },
      "id": 66,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 5
          },
          "id": 64,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaReadDir_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_ops_{{instance}}",
              "refId": "E"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaReadDir",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 5
          },
          "id": 157,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_count_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_count_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_count_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_count_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_count_{{instance}}",
              "refId": "L"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaCreateInode",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 5
          },
          "id": 156,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_count_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaDeleteInode_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_count_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_count_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_count_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_count_{{instance}}",
              "refId": "L"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "DeleteInode",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 12
          },
          "id": 160,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteDentry{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaDeleteDentry_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaDeleteDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 12
          },
          "id": 68,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateDentry{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaCreateDentry_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaCreateDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 12
          },
          "id": 102,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "rate([[app]]_fuseclient_StreamWrite_count{app=\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamWrite_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_StreamRead_count{app=\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_StreamSyncWrite_count{instance=\"$instance\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "SyncWrite",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_StreamSyncOverwrite_count{instance=\"$instance\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "SyncOverwrite",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_fuseclient_StreamOverwrite_count{instance=\"$instance\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Overwrite",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaExtentsList_count{app=\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsList_count_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaExtentsList{app=\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsList_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaExtentsList",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 19
          },
          "id": 69,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaExtentsAdd{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsAdd_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaExtentsAdd_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsAdd_count_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaExtentsList{instance=~\"$instance\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetExtents",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaExtentsAdd",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 19
          },
          "id": 148,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_ops_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_ops_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_ops_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaLookup_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_ops_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_ops_{{instance}}",
              "refId": "H"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaLookup",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 19
          },
          "id": 67,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_count_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_count_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_count_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_count_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_count_{{instance}}",
              "refId": "L"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaBatchInodeGet",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 26
          },
          "id": 152,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_filecreate{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "filecreate_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_filecreate_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "filecreate_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_filesync{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filesync_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_filesync_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filesync_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_filewrite{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filewrite_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_filewrite_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filewrite_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "FileCreate",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 26
          },
          "id": 158,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaInodeGet_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_count_{{instance}}",
              "refId": "K"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "InodeGet",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 26
          },
          "id": 155,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_count_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_count_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaEvictInode_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_count_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_count_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_count_{{instance}}",
              "refId": "L"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "EvictInode",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 33
          },
          "id": 153,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_filecreate{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filecreate_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_filecreate_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filecreate_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_filesync{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "filesync_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_filesync_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "filesync_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_filewrite{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filewrite_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_filewrite_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filewrite_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "FileSync",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 33
          },
          "id": 100,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_fileread{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "fileread_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_fileread_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "fileread_count_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "FileRead",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 33
          },
          "id": 154,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_filecreate{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filecreate_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_filecreate_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filecreate_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_filesync{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filesync_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_filesync_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filesync_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_filewrite{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "filewrite_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_filewrite_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "filewrite_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "FileWrite",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 40
          },
          "id": 147,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_ops_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_ops_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_ops_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_ops_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_ops_{{instance}}",
              "refId": "H"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaTruncate",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 40
          },
          "id": 149,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_ops_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_ops_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaSetattr_cout{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_ops_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaLookup_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_ops_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_ops_{{instance}}",
              "refId": "H"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaSetattr",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 40
          },
          "id": 150,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_ops_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_ops_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_ops_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_ops_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_ops_{{instance}}",
              "refId": "H"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaOpen",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 47
          },
          "id": 106,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_FlushInRead{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "FlushInRead_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_StreamPrepareReadRequests{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "PrepareReadRequest_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "FlushInRead",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 47
          },
          "id": 161,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaUpdateDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 47
          },
          "id": 98,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_StreamWrite{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "Write",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_StreamRead{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "Read",
              "refId": "C"
            },
            {
              "expr": "[[app]]_fuseclient_StreamOverwrite{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "Overwrite",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_StreamCreateExtent{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "CreateExtent",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_StreamSyncWrite{instance=~\"$instance\",cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "SyncWrite",
              "refId": "E"
            },
            {
              "expr": "[[app]]_fuseclient_StreamSyncOverwrite{instance=~\"$instance\",cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "SyncOverwrite",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "StreamSDKLatency",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 54
          },
          "id": 159,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "Truncate_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "Truncate_count_{{instance}}",
              "refId": "L"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "Truncate",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 54
          },
          "id": 151,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_FlushInRead{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "FlushInRead_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_PrepareReadRequest{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "PrepareReadRequest_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "PrepareReadRequest",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        }
      ],
      "title": "FuseClient",
      "type": "row"
    },
    {
      "collapsed": true,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 53
      },
      "id": 60,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 0,
            "y": 2
          },
          "id": 61,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_goroutines{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "go_goroutines_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_threads{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "go_threads_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "go_goroutines",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 7,
            "y": 2
          },
          "id": 169,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_goroutines{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "go_goroutines_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_threads{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "go_threads_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "go_threads",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 14,
            "y": 2
          },
          "id": 63,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_gc_duration_seconds{instance=~\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "seconds_{{quantile}}",
              "refId": "A"
            },
            {
              "expr": "rate(go_gc_duration_seconds_count{instance=~\"$instance\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "gc_rate",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "gc_duration",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "s",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 0,
            "y": 8
          },
          "id": 109,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "process_resident_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\" }",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "resident_memory_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "process_virtual_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_bytes_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "process_virtual_memory_max_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_max_bytes_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "process_resident_memory",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 7,
            "y": 8
          },
          "id": 166,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "process_resident_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\" }",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "resident_memory_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "process_virtual_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_bytes_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "process_virtual_memory_max_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_max_bytes_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "process_virtual_memory",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 14,
            "y": 8
          },
          "id": 167,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "process_resident_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\" }",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "resident_memory_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "process_virtual_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_bytes_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "process_virtual_memory_max_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_max_bytes_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "process_virtual_memory_max",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 0,
            "y": 14
          },
          "id": 110,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "process_open_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "open_fds_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "process_max_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "max_fds_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "open_fds",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 7,
            "y": 14
          },
          "id": 168,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "process_open_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "open_fds_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "process_max_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "max_fds_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "max_fds",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 14,
            "y": 14
          },
          "id": 111,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "rate(process_cpu_seconds_total{cluster=\"$cluster\", role=\"$role\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "cpu_seconds_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "process_cpu",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "s",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 0,
            "y": 20
          },
          "id": 62,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "alloc_bytes",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 7,
            "y": 20
          },
          "id": 172,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "heap_alloc",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 14,
            "y": 20
          },
          "id": 171,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "heap_inuse",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 0,
            "y": 26
          },
          "id": 170,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "sys_bytes",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 7,
            "y": 26
          },
          "id": 173,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "other_sys",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 14,
            "y": 26
          },
          "id": 174,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "mcache_sys",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        }
      ],
      "title": "GoRuntime",
      "type": "row"
    }
  ],
  "refresh": false,
  "schemaVersion": 16,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "allValue": null,
        "current": {
          "selected": true,
          "text": "cfs",
          "value": "cfs"
        },
        "hide": 2,
        "includeAll": false,
        "label": "App",
        "multi": false,
        "name": "app",
        "options": [
          {
            "selected": true,
            "text": "cfs",
            "value": "cfs"
          }
        ],
        "query": "cfs",
        "type": "custom"
      },
      {
        "allValue": null,
        "current": {
          "tags": [],
          "text": "spark",
          "value": "spark"
        },
        "datasource": "default",
        "hide": 0,
        "includeAll": false,
        "label": "Cluster",
        "multi": false,
        "name": "cluster",
        "options": [],
        "query": "label_values(go_info{app=~\"$app\"}, cluster)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "text": "fuseclient",
          "value": "fuseclient"
        },
        "datasource": "default",
        "hide": 0,
        "includeAll": false,
        "label": "Role",
        "multi": false,
        "name": "role",
        "options": [],
        "query": "label_values(go_info{app=~\"$app\", cluster=~\"$cluster\"}, role)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "text": "10.170.6.247:9500",
          "value": "10.170.6.247:9500"
        },
        "datasource": "default",
        "hide": 0,
        "includeAll": false,
        "label": "Instance",
        "multi": false,
        "name": "instance",
        "options": [],
        "query": "label_values(go_info{app=~\"$app\", role=~\"$role\", cluster=~\"$cluster\"}, instance)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "text": "10.170.6.247",
          "value": "10.170.6.247"
        },
        "datasource": "default",
        "hide": 2,
        "includeAll": false,
        "label": "Host",
        "multi": false,
        "name": "hostip",
        "options": [],
        "query": "label_values(go_info{instance=~\"$instance\", cluster=~\"$cluster\"}, instance)",
        "refresh": 1,
        "regex": "/([^:]+):.*/",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "tags": [],
          "text": "clickhourse",
          "value": "clickhourse"
        },
        "datasource": "default",
        "hide": 0,
        "includeAll": false,
        "label": "Vol",
        "multi": false,
        "name": "vol",
        "options": [],
        "query": "label_values(cfs_master_vol_total_GB{app=\"$app\",cluster=\"$cluster\"}, volName)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      }
    ]
  },
  "time": {
    "from": "now-1h",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "",
  "title": "chubaofs",
  "uid": "J8XJyOmZk",
  "version": 484
}

Tune FUSE Performance

Fetch Linux kernel source code

Download the corresponding src rpm, and use the following commands to install source code.

rpm -i kernel-3.10.0-327.28.3.el7.src.rpm 2>&1 | grep -v exist
cd ~/rpmbuild/SPECS
rpmbuild -bp --target=$(uname -m) kernel.spec

The source code will be installed in ~/rpmbuild/BUILD/

Optimize FUSE linux kernel module

In order to achieve maximum throughput performance, several FUSE kernel parameters have to be modified, such as FUSE_MAX_PAGES_PER_REQ and FUSE_DEFAULT_MAX_BACKGROUND.

Update source code according to the following lines.

/* fs/fuse/fuse_i.h */
#define FUSE_MAX_PAGES_PER_REQ 256

/* fs/fuse/inode.c */
#define FUSE_DEFAULT_MAX_BACKGROUND 32

Build against current running Linux kernel

yum install kernel-devel-3.10.0-327.28.3.el7.x86_64

cd ~/rpmbuild/BUILD/kernel-3.10.0-327.28.3.el7/linux-3.10.0-327.28.3.el7.x86_64/fs/fuse
make -C /lib/modules/`uname -r`/build M=$PWD

Install kernel module

cp fuse.ko /lib/modules/`uname -r`/kernel/fs/fuse

rmmod fuse
depmod -a
modprobe fuse

Run Cluster on Docker

Under the docker directory, a helper tool called run_docker.sh is provided to run ChubaoFS with docker-compose.

To start a minimal ChubaoFS cluster from scratch, note that /data/disk is arbitrary, and make sure there are at least 30G available space.

$ docker/run_docker.sh -r -d /data/disk

If client starts successfully, use mount command in client docker shell to check mount status:

$ mount | grep chubaofs

Open http://127.0.0.1:3000 in browser, login with admin/123456 to view grafana monitor metrics.

Or run server and client seperately by following commands:

$ docker/run_docker.sh -b
$ docker/run_docker.sh -s -d /data/disk
$ docker/run_docker.sh -c
$ docker/run_docker.sh -m

For more usage:

$ docker/run_docker.sh -h

Prometheus and Grafana confg can be found in docker/monitor directory.

Resource Manager (Master) API

Cluster

Overview

curl -v "http://10.196.59.198:17010/admin/getCluster" | python -m json.tool

Display the base information of the cluster, such as the detail of metaNode, dataNode, vol and so on.

response

{
    "Name": "test",
    "LeaderAddr": "10.196.59.198:17010",
    "DisableAutoAlloc": false,
    "Applied": 225,
    "MaxDataPartitionID": 100,
    "MaxMetaNodeID": 3,
    "MaxMetaPartitionID": 1,
    "DataNodeStatInfo": {},
    "MetaNodeStatInfo": {},
    "VolStatInfo": {},
    "BadPartitionIDs": {},
    "BadMetaPartitionIDs": {},
    "MetaNodes": {},
    "DataNodes": {}
}

Freeze

curl -v "http://10.196.59.198:17010/cluster/freeze?enable=true"

If cluster is freezed, the vol never allocates dataPartitions automatically.

Parameters

Parameter	Type	Description
enable	bool	if enable is true, the cluster is freezed

Statistics

curl -v "http://10.196.59.198:17010/cluster/stat"

Show cluster space information by zone.

response

{
    "DataNodeStatInfo": {
        "TotalGB": 1,
        "UsedGB": 0,
        "IncreasedGB": -2,
        "UsedRatio": "0.0"
    },
    "MetaNodeStatInfo": {
        "TotalGB": 1,
        "UsedGB": 0,
        "IncreasedGB": -8,
        "UsedRatio": "0.0"
    },
    "ZoneStatInfo": {
        "zone1": {
            "DataNodeStat": {
                "TotalGB": 1,
                "UsedGB": 0,
                "AvailGB": 0,
                "UsedRatio": 0,
                "TotalNodes": 0,
                "WritableNodes": 0
            },
            "MetaNodeStat": {
                "TotalGB": 1,
                "UsedGB": 0,
                "AvailGB": 0,
                "UsedRatio": 0,
                "TotalNodes": 0,
                "WritableNodes": 0
            }
        }
    }
}

Topology

curl -v "http://10.196.59.198:17010/topo/get"

Show cluster topology information by zone.

response

[
    {
        "Name": "zone1",
        "Status": "available",
        "NodeSet": {
            "700": {
                "DataNodeLen": 0,
                "MetaNodeLen": 0,
                "MetaNodes": [],
                "DataNodes": []
            }
        }
    },
    {
        "Name": "zone2",
        "Status": "available",
        "NodeSet": {
            "800": {
                "DataNodeLen": 0,
                "MetaNodeLen": 0,
                "MetaNodes": [],
                "DataNodes": []
            }
        }
    }
]

Update Zone

curl -v "http://10.196.59.198:17010/zone/update?name=zone1&enable=false"

Set the status of the zone to available or unavailable.

Parameters

Parameter	Type	Description
name	string	zone name
enable	bool	if enable is true, the cluster is available

Get Zone

curl -v "http://10.196.59.198:17010/zone/list"

Get name and status of all zones.

response

[
    {
        "Name": "zone1",
        "Status": "available",
        "NodeSet": {}
    },
    {
        "Name": "zone2",
        "Status": "available",
        "NodeSet": {}
    }
]

Get Node Info

curl -v "http://192.168.0.11:17010/admin/getNodeInfo"

Get node info of cluster.

response

{
    "code": 0,
    "msg": "success",
    "data": {
        "batchCount": 0,
        "deleteWorkerSleepMs": 0,
        "markDeleteRate": 0
    }
}

Set Node Info

curl -v "http://192.168.0.11:17010/admin/setNodeInfo?batchCount=100&markDeleteRate=100&deleteWorkerSleepMs=1000"

Set node info of cluster.

Parameters

Parameter	Type	Description
batchCount	uint64	metanode delete batch count
deleteWorkerSleepMs	uint64	metanode delete worker sleep time with millisecond. if 0 for no sleep
markDeleteRate	uint64	datanode batch markdelete limit rate. if 0 for no infinity limit

Metanode Related

GET

curl -v "http://10.196.59.198:17010/metaNode/get?addr=10.196.59.202:17210"  | python -m json.tool

Show the base information of the metaNode, such as addr, total memory, used memory and so on.

Parameters

Parameter	Type	Description
addr	string	the addr which communicate with master

response

{
    "ID": 3,
    "Addr": "10.196.59.202:17210",
    "IsActive": true,
    "Zone": "zone1",
    "MaxMemAvailWeight": 66556215048,
    "TotalWeight": 67132641280,
    "UsedWeight": 576426232,
    "Ratio": 0.008586377967698518,
    "SelectCount": 0,
    "Carry": 0.6645600532184904,
    "Threshold": 0.75,
    "ReportTime": "2018-12-05T17:26:28.29309577+08:00",
    "MetaPartitionCount": 1,
    "NodeSetID": 2,
    "PersistenceMetaPartitions": {}
}

Decommission

curl -v "http://127.0.0.1/metaNode/decommission?addr=127.0.0.1:9021"

Remove the metaNode from cluster, meta partitions which locate the metaNode will be migrate other available metaNode asynchronous.

Parameters

Parameter	Type	Description
addr	string	the addr which communicate with master

Threshold

curl -v "http://127.0.0.1/threshold/set?threshold=0.75"

If the used memory percent arrives the threshold, the status of the meta partitions which locate the metaNode will be read only.

Parameters

Parameter	Type	Description
threshold	float64	the max percent of memory which metaNode can use

Datanode Related

GET

curl -v "http://10.196.59.198:17010/dataNode/get?addr=10.196.59.201:17310"  | python -m json.tool

Show the base information of the dataNode, such as addr, disk total size, disk used size and so on.

Parameters

Parameter	Type	Description
addr	string	the addr which communicate with master

response

{
    "TotalWeight": 39666212700160,
    "UsedWeight": 2438143586304,
    "AvailableSpace": 37228069113856,
    "ID": 2,
    "Zone": "zone1",
    "Addr": "10.196.59.201:17310",
    "ReportTime": "2018-12-06T10:56:38.881784447+08:00",
    "IsActive": true
    "UsageRatio": 0.06146650815226848,
    "SelectTimes": 5,
    "Carry": 1.0655859145960367,
    "DataPartitionReports": {},
    "DataPartitionCount": 21,
    "NodeSetID": 3,
    "PersistenceDataPartitions": {},
    "BadDisks": {}
}

Decommission

curl -v "http://10.196.59.198:17010/dataNode/decommission?addr=10.196.59.201:17310"

Remove the dataNode from cluster, data partitions which locate the dataNode will be migrate other available dataNode asynchronous.

Parameters

Parameter	Type	Description
addr	string	the addr which communicate with master

Volume

Create

curl -v "http://10.196.59.198:17010/admin/createVol?name=test&capacity=100&owner=cfs&mpCount=3"

Allocate a set of data partition and a meta partition to the user.

Default create 10 data partition and 3 meta partition when create volume.

ChubaoFS uses the Owner parameter as the user ID. When creating a volume, if there is no user named the owner of the volume, a user with the user ID same as Owner will be automatically created; if a user named Owner already exists in the cluster, the volume will be owned by the user. For details, please see:: doc: /admin-api/master/user

Parameters

Parameter	Type	Description	Mandatory	Default
name	string	volume name	Yes	None
capacity	int	the quota of vol, unit is GB	Yes	None
owner	string	the owner of vol, and user ID of a user	Yes	None
mpCount	int	the amount of initial meta partitions	No	3
enableToken	bool	whether to enable the token mechanism to control client permissions	No	false
size	int	the size of data partitions, unit is GB	No	120
followerRead	bool	enable read from follower	No	false
crossZone	bool	cross zone or not. If it is true, parameter zoneName must be empty	No	false
zoneName	string	specified zone	No	default (if crossZone is false)

Delete

curl -v "http://10.196.59.198:17010/vol/delete?name=test&authKey=md5(owner)"

Mark the vol status to MarkDelete first, then delete data partition and meta partition asynchronous, finally delete meta data from persist store.

While deleting the volume, the policy information related to the volume will be deleted from all user information.

Parameters

Parameter	Type	Description
name	string	volume name
authKey	string	calculates the 32-bit MD5 value of the owner field as authentication information

Get

curl -v "http://10.196.59.198:17010/client/vol?name=test&authKey=md5(owner)" | python -m json.tool

Show the base information of the vol, such as name, the detail of data partitions and meta partitions and so on.

Parameters

Parameter	Type	Description
name	string	volume name
authKey	string	calculates the 32-bit MD5 value of the owner field as authentication information

response

{
    "Name": "test",
    "Owner": "user",
    "Status": "0",
    "FollowerRead": "true",
    "MetaPartitions": {},
    "DataPartitions": {},
    "DataPartitions": {},
    "CreateTime": 0
}

Stat

curl -v http://10.196.59.198:17010/client/volStat?name=test

Show the status information of volume.

Parameters

Parameter	Type	Description
name	string	volume name

response

{
    "Name": "test",
    "TotalSize": 322122547200000000,
    "UsedSize": 155515112832780000,
    "UsedRatio": "0.48",
    "EnableToken": false
}

Update

curl -v "http://10.196.59.198:17010/vol/update?name=test&capacity=100&authKey=md5(owner)"

Increase the quota of volume, or adjust other parameters.

Parameters

Parameter	Type	Description	Mandatory
name	string	volume name	Yes
authKey	string	calculates the 32-bit MD5 value of the owner field as authentication information	Yes
capacity	int	the quota of vol, unit is GB	Yes
zoneName	string	update zone name	Yes
enableToken	bool	whether to enable the token mechanism to control client permissions. False by default.	No
followerRead	bool	enable read from follower	No

List

curl -v "http://10.196.59.198:17010/vol/list?keywords=test"

List all volumes information, and can be filtered by keywords.

Parameters

Parameter	Type	Description	Mandatory
keywords	string	get volumes information which contains this keyword	No

response

[
   {
       "Name": "test1",
       "Owner": "cfs",
       "CreateTime": 0,
       "Status": 0,
       "TotalSize": 155515112832780000,
       "UsedSize": 155515112832780000
   },
   {
       "Name": "test2",
       "Owner": "cfs",
       "CreateTime": 0,
       "Status": 0,
       "TotalSize": 155515112832780000,
       "UsedSize": 155515112832780000
   }
]

Add Token

curl -v "http://10.196.59.198:17010/token/add?name=test&tokenType=1&authKey=md5(owner)"

Add the token that controls read and write permissions.

Parameters

Parameter	Type	Description
name	string	the name of vol
tokenType	int	1 is readonly token, 2 is readWrite token
authKey	string	calculates the 32-bit MD5 value of the owner field as authentication information

Update Token

curl -v "http://10.196.59.198:17010/token/update?name=test&token=xx&tokenType=1&authKey=md5(owner)"

Update token type.

Parameters

Parameter	Type	Description
name	string	the name of vol
token	string	the token value
tokenType	int	1 is readonly token, 2 is readWrite token
authKey	string	calculates the 32-bit MD5 value of the owner field as authentication information

Delete Token

curl -v "http://10.196.59.198:17010/token/delete?name=test&token=xx&authKey=md5(owner)"

Delete specified token.

Parameters

Parameter	Type	Description
name	string	the name of vol
token	string	the token value
authKey	string	calculates the 32-bit MD5 value of the owner field as authentication information

Get Token

curl -v "http://10.196.59.198:17010/token/get?name=test&token=xx"

Show token information.

Parameters

Parameter	Type	Description
name	string	the name of vol
token	string	the token value

response

{
    "TokenType":2,
    "Value":"siBtuF9hbnNqXzJfMTU48si3nzU4MzE1Njk5MDM1NQ==",
    "VolName":"test"
}

Meta Partition

Create

curl -v "http://10.196.59.198:17010/metaPartition/create?name=test&start=10000"

Split meta partition manually. If max meta partition of the vol which range is [0,end) and end larger than start parameter, old meta partition range will be [0,start], new meta partition will be [start+1,end).

Parameters

Parameter	Type	Description
name	string	the name of vol
start	uint64	the start value of meta partition which will be create

Get

curl -v "http://10.196.59.198:17010/metaPartition/get?id=1" | python -m json.tool

Show base information of meta partition, such as id, start, end and so on.

Parameters

Parameter	Type	Description
id	uint64	the id of meta partition

response

{
    "PartitionID": 1,
    "Start": 0,
    "End": 9223372036854776000,
    "MaxNodeID": 1,
    "VolName": "test",
    "Replicas": {},
    "ReplicaNum": 3,
    "Status": 2,
    "IsRecover": true,
    "Hosts": {},
    "Peers": {},
    "Zones": {},
    "MissNodes": {},
    "LoadResponse": {}
}

Decommission

curl -v "http://10.196.59.198:17010/metaPartition/decommission?id=13&addr=10.196.59.202:17210"

Remove the replica of meta partition, and create new replica asynchronous.

Parameters

Parameter	Type	Description
id	uint64	the id of meta partition
addr	string	the addr of replica which will be decommission

Load

curl -v "http://10.196.59.198:17010/metaPartition/load?id=1"

Send load task to the metaNode which meta partition locate on, then check the crc of each replica in the meta partition.

Parameters

Parameter	Type	Description
id	uint64	the id of data partition

Data Partition

Create

curl -v "http://10.196.59.198:17010/dataPartition/create?count=400&name=test"

Create a set of data partition.

Parameters

Parameter	Type	Description
count	int	the num of dataPartitions will be create
name	string	the name of vol

Get

curl -v "http://10.196.59.198:17010/dataPartition/get?id=100"  | python -m json.tool

Get information of the specified data partition.

Parameters

Parameter	Type	Description
id	uint64	the id of data partition

response

{
    "PartitionID": 100,
    "LastLoadedTime": 1544082851,
    "ReplicaNum": 3,
    "Status": 2,
    "Replicas": {},
    "Hosts": {},
    "Peers": {},
    "Zones": {},
    "MissingNodes": {},
    "VolName": "test",
    "VolID": 2,
    "FileInCoreMap": {},
    "FilesWithMissingReplica": {}
}

Decommission

curl -v "http://10.196.59.198:17010/dataPartition/decommission?id=13&addr=10.196.59.201:17310"

Remove the replica of data partition, and create new replica asynchronous.

Parameters

Parameter	Type	Description
id	uint64	the id of data partition
addr	string	the addr of replica which will be decommission

Load

curl -v "http://10.196.59.198:17010/dataPartition/load?id=1"

Send load task to the dataNode which data partition locate on, then check the crc of each file in the data partition asynchronous.

Parameters

Parameter	Type	Description
id	uint64	the id of data partition

Offline Disk

curl -v "http://10.196.59.198:17010/disk/decommission?addr=10.196.59.201:17310&disk=/cfs1"

Synchronously offline all the data partitions on the disk, and create a new replica for each data partition in the cluster.

Parameters

Parameter	Type	Description
addr	string	replica address
disk	string	disk path

Master Management

Add

curl -v "http://10.196.59.198:17010/raftNode/add?addr=10.196.59.197:17010&id=3"

Add a new master node to master raft group.

Parameters

Parameter	Type	Description
addr	string	the addr of master server, format is ip:port
id	uint64	the node id of master server

Remove

curl -v "http://10.196.59.198:17010/raftNode/remove?addr=10.196.59.197:17010&id=3"

Remove the master node from master raft group.

Parameters

Parameter	Type	Description
addr	string	the addr of master server, format is ip:port
id	uint64	the node id of master server

User

Create

curl -H "Content-Type:application/json" -X POST --data '{"id":"testuser","pwd":"12345","type":3}' "http://10.196.59.198:17010/user/create"

Create a user in the cluster to access object storage service. When the cluster starts, the root user is automatically created (the value of type is 0x1).

ChubaoFS treats every Owner of volume as a user. For example, if the value of Owner is testuser when creating a volume, the volume is owned by user testuser.

If there is no user ID with the same value as the Owner when creating the volume, the user named the value of Owner will be automatically created when creating the volume.

body key

Key	Type	Description	Range	Mandatory	Default
id	string	user ID	Consists of letters, numbers and underscores, no more than 20 characters	Yes	None
pwd	string	user’s password	Unlimited	No	ChubaoFSUser
ak	string	Access Key	Consists of 16-bits letters and numbers	No	Random value
sk	string	Secret Key	Consists of 32-bits letters and numbers	No	Random value
type	int	user type	2: [admin] / 3: [normal user]	Yes	None

Delete

curl -v "http://10.196.59.198:17010/user/delete?user=testuser"

Delete the specified user in the cluster.

Parameters

Parameter	Type	Description
user	string	user ID

Get

Show basic user information, including user ID, Access Key, Secret Key, list of owned volumes, list of permissions granted by other users, user type, creation time.

The field policy shows the volumes which the user has permission, of which own_vols indicates that volumes owned by the user, and authorized_vols indicates the volume authorized by other users to the user with restrictions.

There are two ways to obtain:

Query by User ID

curl -v "http://10.196.59.198:17010/user/info?user=testuser" | python -m json.tool

Parameters

Parameter	Type	Description
user	string	user ID

Query by Access Key

curl -v "http://10.196.59.198:17010/user/akInfo?ak=0123456789123456" | python -m json.tool

Parameters

Parameter	Type	Description
ak	string	Access Key

response

{
     "user_id": "testuser",
     "access_key": "0123456789123456",
     "secret_key": "ZVY5RHlrnOrCjImW9S3MajtYZyxSegcf",
     "policy": {
         "own_vols": ["vol1"],
         "authorized_vols": {
             "ltptest": [
                 "perm:builtin:ReadOnly",
                 "perm:custom:PutObjectAction"
             ]
         }
     },
     "user_type": 3,
     "create_time": "2020-05-11 09:25:04"
}

List Users

curl -v "http://10.196.59.198:17010/user/list?keywords=test" | python -m json.tool

Query information about all users in a cluster whose user ID contains the keyword.

Parameters

Parameter	Type	Description
keywords	string	check user ID contains this or not

Update

curl -H "Content-Type:application/json" -X POST --data '{"user_id":"testuser","access_key":"KzuIVYCFqvu0b3Rd","secret_key":"iaawlCchJeeuGSnmFW72J2oDqLlSqvA5","type":3}' "http://10.196.59.198:17010/user/update"

Update the specified user’s information, including access key, secret key and user type.

body key

Key	Type	Description	Mandatory
user_id	string	user ID value after updating	Yes
access_key	string	Access Key value after updating	No
secret_key	string	Secret Key value after updating	No
type	int	user type value after updating	No

Update Permission

curl -H "Content-Type:application/json" -X POST --data '{"user_id":"testuser","volume":"vol","policy":["perm:builtin:ReadOnly","perm:custom:PutObjectAction"]}' "http://10.196.59.198:17010/user/updatePolicy"

Update the specified user’s permission to a volume. There are three types of values for policy:

• Grant read-only or read-write permission, the value is perm:builtin:ReadOnly or perm:builtin:Writable.
• Grant a permission of the specified action, the format is action:oss:XXX, take GetObject action as an example, the value of policy is action:oss:GetObject.
• Grant a custom permission, the format is perm:custom:XXX, where XXX is customized by the user.

After the permissions are specified, the user can only access the volume within the specified permissions when using the object storage. If the user already has permissions for this volume, this operation will overwrite the original permissions.

body key

Key	Type	Description	Mandatory
user_id	string	user ID to be set	Yes
volume	string	volume name to be set	Yes
policy	string slice	policy to be set	Yes

Remove Permission

curl -H "Content-Type:application/json" -X POST --data '{"user_id":"testuser","volume":"vol"}' "http://10.196.59.198:17010/user/removePolicy"

Remove all permissions of a specified user for a volume.

body key

Key	Type	Description	Mandatory
user_id	string	user ID to be deleted	Yes
volume	string	volume name to be deleted	Yes

Transfer Volume

curl -H "Content-Type:application/json" -X POST --data '{"volume":"vol","user_src":"user1","user_dst":"user2","force":"true"}' "http://10.196.59.198:17010/user/transferVol"

Transfer the ownership of the specified volume. This operation removes the specified volume from the owner_vols of source user name and adds it to the owner_vols of target user name; At the same time, the value of the field Owner in the volume structure will also be updated to the target user ID.

body key

Key	Type	Description	Mandatory
volume	string	Volume name to be transfered	Yes
user_src	string	Original owner of the volume, and must be the same as the Owner of the volume	Yes
user_dst	string	Target user ID after transferring	Yes
force	bool	Force to transfer the volume. If the value is set to true, even if the value of user_src is different from the value of the owner of the volume, the volume will also be transferred to the target user	No

Meta Node API

Meta Partition

Get Partitions

curl -v  http://10.196.59.202:17210/getPartitions

Get all meta-partition base information of the metanode.

Get Partition by ID

curl -v http://10.196.59.202:17210/getPartitionById?pid=100

Get the specified partition information, this result contains: leader address, raft group peer and cursor.

Parameters

Parameter	Type	Description
pid	integer	meta-partition id

Inode

Get Inode

curl -v http://10.196.59.202:17210/getInode?pid=100&ino=1024

Get inode information

Parameters

Parameter	Type	Description
pid	integer	meta-partition id
ino	integer	inode id

Get Extents by Inode

curl -v http://10.196.59.202:17210/getExtentsByInode?pid=100&ino=1024

Get inode all extents information

Parameters

Parameter	Type	Description
pid	integer	meta-partition id
ino	integer	inode id

Get All Inodes

curl -v http://10.196.59.202:17210/getAllInodes?pid=100

Get all inodes of the specified partition

Parameters

Parameter	Type	Description
pid	integer	meta-partition id

Dentry

Get Dentry

curl -v 'http://10.196.59.202:17210/getDentry?pid=100&name=""&parentIno=1024'

Get dentry information

Parameters

Parameter	Type	Description
pid	integer	meta partition id
name	string	file or directory name
parentIno	integer	file or directory parent directory inode

Get Directory

curl -v "http://10.196.59.202:17210/getDirectory?pid=100&parentIno=1024"

Get all files of the parent inode is 1024

Parameters

Parameter	Type	Description
pid	integer	partition id
ino	integer	inode id

Get All Dentry

curl -v "http://10.196.59.202:17210/getAllDentry?pid=100"

Parameters

Parameter	Type	Description
pid	integer	partition id

Command Line Interface

CLI

Using the Command Line Interface (CLI) can manage the cluster conveniently. With this tool, you can view the status of the cluster and each node, and manage each node, volumes, or users.

As the CLI continues to improve, it will eventually achieve 100% coverage of the APIs of each node in the cluster.

Compile and Configure

After downloading the ChubaoFS source code, execute the command go build in the directory chubaofs/cli to generate cli.

At the same time, a configuration file named .cfs-cli.json will be generated in the directory root, and the master address can be changed to the current cluster master address. You can also get or set the master address by executing the command ./cli config info or ./cli config set.

Bug Shooting

The logs of cfs-cli tool are in the directory /tmp/cfs/cli, which offer detail running information for bug shooting.

Usage

In the directory chubaofs/cli, execute the command ./cli --help or ./cli -h to get the CLI help document.

CLI is mainly divided into seven types of management commands.

Commands List

Command	description
cli cluster	Manage cluster components
cli metanode	Manage meta nodes
cli datanode	Manage data nodes
cli datapartition	Manage data partitions
cli metapartition	Manage meta partitions
cli config	Manage configuration for cli tool
cli completion	Generating bash completions
cli volume, vol	Manage cluster volumes
cli user	Manage cluster users
cli compatibility	Compatibility test

Cluster Management

./cli cluster info          #Show cluster summary information

./cli cluster stat          #Show cluster status information

./cli cluster freeze [true/false]        #Turn on or turn off the automatic allocation of the data partitions.

./cli cluster threshold [float]     #Set the threshold of memory on each meta node.

MetaNode Management

./cli metanode list          #List information of meta nodes

./cli metanode info [Address]     #Show detail information of a meta node

./cli metanode decommission [Address] #Decommission partitions in a meta node to other nodes

DataNode Management

./cli datanode list          #List information of data nodes

./cli datanode info [Address]         #Show detail information of a data node

./cli datanode decommission [Address]   #Decommission partitions in a data node to other nodes

DataPartition Management

./cli datapartition info [Partition ID]        #Display detail information of a data partition

./cli datapartition decommission [Address] [Partition ID]   #Decommission a replication of the data partition to a new address

./cli datapartition add-replica [Address] [Partition ID]    #Add a replication of the data partition on a new address

./cli datapartition del-replica [Address] [Partition ID]    #Delete a replication of the data partition from a fixed address

./cli datapartition check    #Diagnose partitions, display the partitions those are corrupt or lack of replicas

MetaPartition Management

./cli metapartition info [Partition ID]        #Display detail information of a meta partition

./cli metapartition decommission [Address] [Partition ID]   #Decommission a replication of the meta partition to a new address

./cli metapartition add-replica [Address] [Partition ID]    #Add a replication of the meta partition on a new address

./cli metapartition del-replica [Address] [Partition ID]    #Delete a replication of the meta partition from a fixed address

./cli metapartition check    #Diagnose partitions, display the partitions those are corrupt or lack of replicas

Config Management

./cli config info     #Show configurations of cli

./cli config set [flags]    #Set configurations of cli
Flags:
    --addr      string      #Specify master address [{HOST}:{PORT}]
    --timeout   uint16      #Specify timeout for requests [Unit: s] (default 60)

Completion Management

./cli completion      #Generate bash completions

Volume Management

./cli volume create [VOLUME NAME] [USER ID] [flags]     #Create a new volume
Flags:
    --capacity uint                                     #Specify volume capacity [Unit: GB] (default 10)
    --dp-size  uint                                     #Specify size of data partition size [Unit: GB] (default 120)
    --follower-read                                     #Enable read form replica follower (default true)
    --mp-count int                                      #Specify init meta partition count (default 3)
    -y, --yes                                           #Answer yes for all questions

./cli volume delete [VOLUME NAME] [flags]               #Delete a volume from cluster
Flags:
    -y, --yes                                           #Answer yes for all questions

./cli volume info [VOLUME NAME] [flags]                 #Show volume information
Flags:
    -d, --data-partition                                #Display data partition detail information
    -m, --meta-partition                                #Display meta partition detail information

./cli volume add-dp [VOLUME] [NUMBER]                   #Create and add more data partition to a volume

./cli volume list                                       #List cluster volumes

./cli volume transfer [VOLUME NAME] [USER ID] [flags]   #Transfer volume to another user. (Change owner of volume)
Flags：
    -f, --force                                         #Force transfer without current owner check
    -y, --yes                                           #Answer yes for all questions

User Management

./cli user create [USER ID] [flags]         #Create a new user
Flags：
    --access-key string                     #Specify user access key for object storage interface authentication
    --secret-key string                     #Specify user secret key for object storage interface authentication
    --password string                       #Specify user password
    --user-type string                      #Specify user type [normal | admin] (default "normal")
    -y, --yes                               #Answer yes for all questions

./cli user delete [USER ID] [flags]         #Delete specified user
Flags：
    -y, --yes                               #Answer yes for all questions

./cli user info [USER ID]                   #Show detail information about specified user

./cli user list                             #List cluster users

./cli user perm [USER ID] [VOLUME] [PERM]   #Setup volume permission for a user
                                            #The value of [PERM] is READONLY, RO, READWRITE, RW or NONE

./cli user update [USER ID] [flags]         #Update information about specified user
Flags：
    --access-key string                     #Update user access key
    --secret-key string                     #Update user secret key
    --user-type string                      #Update user type [normal | admin]
    -y, --yes                               #Answer yes for all questions

Compatibility Test

./cli cptest meta [Snapshot Path] [Host] [Partition ID]         #Metadata compatibility test
Parameters：
        [Snapshot Path] string                     #The path which snapshot file located
        [Host] string                              #The metanode host which generated the snapshot file
        [Partition ID] string                      #The meta partition ID which to be compared

Example:

1. Use the old version to prepare metadata, stop writing metadata,after waiting for the latest snapshot to be generated(about 5 minutes), copy the snapshot file to the local machine
2. Execute the metadata comparison command on local machine

[Verify result]
All dentry are consistent
All inodes are consistent
All meta has checked

Use Cases

ChubaoFS is a distributed file system that is compatible with most POSIX file system semantics. When ChubaoFS is mounted, it can be as simple as using a local file system. Basically, it can be used in any case where a file system is needed, replacing the local file system, and realizing infinitely expandable storage without physical boundaries. It has been applied in various scenarios, and the following are some of the extracted scenes.

Machine Learning

Disadvantages of using a local disk to store training data sets:

• The local disk space is small, and there are multiple models. The training data set of each model reaches the TB level. If you use a local disk to store the training data set, you need to reduce the size of the training data set.
• Training data sets need to be updated frequently, requiring more disk space.
• Risk of loss of training data set if machine crashes.

The advantages of using ChubaoFS to store training data sets:

• Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.
• Multiple replicas of data to ensure high data reliability without worrying about losing data.
• Compatible with POSIX file system interface, no changes required by the application.

ElasticSearch

Using local disks to store data often encounters the following problems:

• Disk usage is uneven and disk IO cannot be fully utilized.
• Local disk space is limited.

The advantages of using ChubaoFS as a backend storage:

• Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.
• Disk IO usage is uniform and disk IO is fully utilized.
• Ensure data is highly reliable without worrying about losing data.

Nginx Log Storage

Do you often worry about running out of local disk space? With ChubaoFS, you can store datas in a distributed file system without worrying about running out of disk space. If you use a local disk to store logs, you may often worry about the following issues:

• Docker local disk space is small.
• If the docker container crashes, the log is lost and unrecoverable.
• The mixed deployment of physical machine and docker machine is difficult to manage and has high operation and maintenance cost.

The advantages of using ChubaoFS to store nginx logs are:

• The disk space is unlimited and easy to expand. The disk capacity is automatically expanded according to the percentage of disk usage. The storage system can be expanded on demand, which greatly saves storage costs.
• Ensure data is highly reliable and do not worry about losing data.
• Multiple replicas to solve the problem of unable to write to the log caused by disk error and datanode crashes.
• Compatible with the POSIX file system interface, no changes required by the application.
• The operation and maintenance of ChubaoFS are simple, and one person can easily manage the cluster of tens of thousands of machines.

Spark

In the big data set scenario, are you worried about the amount of data stored in Spark intermediate calculation results in order to carefully calculate each task? You can store the shuffle results to ChubaoFS, and no longer worry about the disk has no free space which causes the task to fail. This enables the separation of storage and computation. The pain points of using local disk to store shuffle intermediate results:

• Insufficient disk space.
• Too many temporary directory files to create new files.

The advantages of using ChubaoFS:

• Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.
• Meta node manages file metadata, which can be expanded horizontally and the number of files is unlimited.

MySQL Database Backup

Disadvantages of using OSS(Object Storage Service) to back up MySQL database:

• Need to use the OSS SDK or RESTful API to develop backup programs which increases the difficulty of operation and maintenance.
• If backup file fails, troubleshooting is more difficult.
• After backing up files to OSS, it is not convenient to check whether the files are successfully uploaded.
• Backup files are processed by multiple layers of services, which affects performance.

Advantages of using ChubaoFS to back up MySQL databases:

• Easy to use, compatible with POSIX file interface, and can be used as a local file system.
• Complete and detailed operation logs are stored in the local file system, making it easy to troubleshoot problems.
• Simply execute the ls command to verify that the file was successfully uploaded.
• Supports PageCache and WriteCache, and file read and write performance is significantly improved compared to OSS.

Performance

Environment

Cluster Information

Instance	Nodes	CPU	Memory	Storage	Network	Description
Master	3	32	32 GB	120 GB SSD	10 Gb/s
MetaNode	10	32	32 GB	16 x 1TB SSD	10 Gb/s	hybrid deployment
DataNode	10	32	32 GB	16 x 1TB SSD	10 Gb/s	hybrid deployment

Volume Setup

Parameter	Default	Recommend	Description
FollowerRead	True	True
Capacity	10 GB	300 000 000 GB
Data Replica Number	3	3
Meta Replica Number	3	3
Data Partition Size	120 GB	120 GB	Logical upper limit with no pre-occupied space.
Data Partition Count	10	1500
Meta Partition Count	3	10
Cross Zone	False	False

Set volume parameters by following:

$ cfs-cli volume create test-vol {owner} --capacity=300000000 --mp-count=10
Create a new volume:
  Name                : test-vol
  Owner               : ltptest
  Dara partition size : 120 GB
  Meta partition count: 10
  Capacity            : 300000000 GB
  Replicas            : 3
  Allow follower read : Enabled

Confirm (yes/no)[yes]: yes
Create volume success.

$ cfs-cli volume add-dp test-vol 1490

client configuration

Parameter	Default	Recommend
rate limit	-1	-1

#get current iops, default:-1(no limits on iops)：
$ http://[ClientIP]:[ProfPort]/rate/get
#set iops
$ http://[ClientIP]:[ProfPort]/rate/set?write=800&read=800

Small File Performance and Scalability

Small file operation performance and scalability benchmark test by mdtest.

Setup

#!/bin/bash
set -e
TARGET_PATH="/mnt/test/mdtest" # mount point of ChubaoFS volume
for FILE_SIZE in 1024 2048 4096 8192 16384 32768 65536 131072 # file size
do
mpirun --allow-run-as-root -np 512 --hostfile hfile64 mdtest -n 1000 -w $i -e $FILE_SIZE -y -u -i 3 -N 1 -F -R -d $TARGET_PATH;
done

Benchmark

File Size (KB)	1	2	4	8	16	32	64	128
Creation (TPS)	70383	70383	73738	74617	69479	67435	47540	27147
Read (TPS)	108600	118193	118346	122975	116374	110795	90462	62082
Removal (TPS)	87648	84651	83532	79279	85498	86523	80946	84441
Stat (TPS)	231961	263270	264207	252309	240244	244906	273576	242930

IO Performance and Scalability

IO Performance and benchmark scalability test by fio.

Note: Multiple clients mount the same volume. And the process refers to the fio process.

1. Sequential Read

Setup

#!/bin/bash
fio -directory={} \
    -ioengine=psync \
    -rw=read \  # sequential read
    -bs=128k \  # block size
    -direct=1 \ # enable direct IO
    -group_reporting=1 \
    -fallocate=none \
    -time_based=1 \
    -runtime=120 \
    -name=test_file_c{} \
    -numjobs={} \
    -nrfiles=1 \
    -size=10G

Bandwidth(MB/s)

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	148.000	626.000	1129.000	1130.000
2 Clients	284.000	1241.000	2258.000	2260.000
4 Clients	619.000	2640.000	4517.000	4515.000
8 Clients	1193.000	4994.000	9006.000	9034.000

IOPS

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	1180.000	5007.000	9031.000	9040.000
2 Clients	2275.000	9924.000	18062.000	18081.000
4 Clients	4954.000	21117.000	36129.000	36112.000
8 Clients	9531.000	39954.000	72048.000	72264.000

Latency(Microsecond)

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	842.200	794.340	1767.310	7074.550
2 Clients	874.255	801.690	1767.370	7071.715
4 Clients	812.363	760.702	1767.710	7077.065
8 Clients	837.707	799.851	1772.620	7076.967

2. Sequential Write

Setup

#!/bin/bash
fio -directory={} \
    -ioengine=psync \
    -rw=write \ # sequential write
    -bs=128k \  # block size
    -direct=1 \ # enable direct IO
    -group_reporting=1 \
    -fallocate=none \
    -name=test_file_c{} \
    -numjobs={} \
    -nrfiles=1 \
    -size=10G

Bandwidth(MB/s)

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	52.200	226.000	956.000	1126.000
2 Clients	104.500	473.000	1763.000	2252.000
4 Clients	225.300	1015.000	2652.000	3472.000
8 Clients	480.600	1753.000	3235.000	3608.000

IOPS

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	417	1805	7651	9004
2 Clients	835	3779	14103	18014
4 Clients	1801	8127	21216	27777
8 Clients	3841	14016	25890	28860

Latency(Microsecond)

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	2385.400	2190.210	2052.360	7081.320
2 Clients	2383.610	2081.850	2233.790	7079.450
4 Clients	2216.305	1947.688	2946.017	8842.903
8 Clients	2073.921	2256.120	4787.496	17002.425

3. Random Read

Setup

#!/bin/bash
fio -directory={} \
    -ioengine=psync \
    -rw=randread \ # random read
    -bs=4k \       # block size
    -direct=1 \    # enable direct IO
    -group_reporting=1 \
    -fallocate=none \
    -time_based=1 \
    -runtime=120 \
    -name=test_file_c{} \
    -numjobs={} \
    -nrfiles=1 \
    -size=10G

Bandwidth(MB/s)

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	6.412	39.100	216.000	534.000
2 Clients	14.525	88.100	409.000	1002.000
4 Clients	33.242	200.200	705.000	1693.000
8 Clients	59.480	328.300	940.000	2369.000

IOPS

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	1641	10240	56524.800	140288
2 Clients	3718	23142.4	107212.8	263168
4 Clients	8508	52428.8	184627.2	443392
8 Clients	15222	85072.8	246681.6	621056

Latency(Microsecond)

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	603.580	395.420	287.510	466.320
2 Clients	532.840	351.815	303.460	497.100
4 Clients	469.025	317.140	355.105	588.847
8 Clients	524.709	382.862	530.811	841.985

4. Random Write

Setup

#!/bin/bash
fio -directory={} \
    -ioengine=psync \
    -rw=randwrite \ # random write
    -bs=4k \        # block size
    -direct=1 \     # enable direct IO
    -group_reporting=1 \
    -fallocate=none \
    -time_based=1 \
    -runtime=120 \
    -name=test_file_c{} \
    -numjobs={} \
    -nrfiles=1 \
    -size=10G

Bandwidth(MB/s)

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	3.620	17.500	118.000	318.000
2 Clients	7.540	44.800	230.000	476.000
4 Clients	16.245	107.700	397.900	636.000
8 Clients	39.274	208.100	487.100	787.100

IOPS

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	926.000	4476.000	31027.200	83251.200
2 Clients	1929.000	11473.000	60313.600	124620.800
4 Clients	4156.000	27800.000	104243.200	167014.400
8 Clients	10050.000	53250.000	127692.800	206745.600

Latency

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	1073.150	887.570	523.820	784.030
2 Clients	1030.010	691.530	539.525	1042.685
4 Clients	955.972	575.183	618.445	1552.205
8 Clients	789.883	598.393	1016.185	2506.424

Metadata Performance and Scalability

Metadata performance and scalability benchmark test by mdtest.

Setup

#!/bin/bash
TEST_PATH=/mnt/cfs/mdtest # mount point of ChubaoFS volume
for CLIENTS in 1 2 4 8 # number of clients
do
mpirun --allow-run-as-root -np $CLIENTS --hostfile hfile01 mdtest -n 5000 -u -z 2 -i 3 -d $TEST_PATH;
done

Dir Creation

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	448.618	2421.001	14597.97	43055.15
2 Clients	956.947	5917.576	28930.431	72388.765
4 Clients	2027.02	13213.403	54449.056	104771.356
8 Clients	4643.755	27416.904	89641.301	119542.62

Dir Removal

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	399.779	2118.005	12351.635	34903.672
2 Clients	833.353	5176.812	24471.674	50242.973
4 Clients	1853.617	11462.927	46413.313	91128.059
8 Clients	4441.435	24133.617	74401.336	115013.557

Dir Stat

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	283232.761	1215309.524	4231088.104	12579177.02
2 Clients	572834.143	2169669.058	8362749.217	18120970.71
4 Clients	1263474.549	3333746.786	10160929.29	31874265.88
8 Clients	2258670.069	8715752.83	22524794.98	77533648.04

File Creation

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	448.888	2400.803	13638.072	27785.947
2 Clients	925.68	5664.166	25889.163	50434.484
4 Clients	2001.137	12986.968	50330.952	91387.825
8 Clients	4479.831	25933.437	86667.966	112746.199

File Removal

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	605.143	3678.138	18631.342	47035.912
2 Clients	1301.151	8365.667	34005.25	64860.041
4 Clients	3032.683	14017.426	50938.926	80692.761
8 Clients	7170.386	32056.959	68761.908	88357.563

Tree Creation

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	305.778	229.562	86.299	23.917
2 Clients	161.31	211.119	76.301	24.439
4 Clients	260.562	223.153	81.209	23.867
8 Clients	350.038	220.744	81.621	17.144

Tree Removal

	1 Process	4 Processes	16 Processes	64 Processes
1 Client	137.462	70.881	31.235	7.057
2 Clients	217.026	113.36	23.971	7.128
4 Clients	231.622	113.539	30.626	7.299
8 Clients	185.156	83.923	20.607	5.515

Integrity

• Linux Test Project / fs

Workload

• Database backup

• Java application logs

• Code git repo

• Database systems

  MyRocks, MySQL Innodb, HBase,

Scalability

• Volume Scalability: tens to millions of cfs volumes
• Metadata Scalability: a big volume with billions of files/directories

Contributing to ChubaoFS

Bug Reports

Please make sure the bug is not already reported by searching the repository with reasonable keywords. Then, open an issue with steps to reproduce.

Patch Guidelines

In order for the patch to be accepted with higher possibility, there are a few things you might want to pay attention to:

• Filesystem stress tests is required before opening a pull request by

runltp -f fs -d [MOUNTPOINT]

• A good commit message describing the bug fix or the new feature is preferred.
• DCO is required, so please add Signed-off-by to the commit.

Credits

Sections of this documents have been borrowed from CoreDNS and Fluentd

Environment Requirements and Capacity Planning

Environment Requirements

The following table lists the system and hardware requirements of the performance test environment and production environment. You can also refer to the capacity planning chapter to accurately customize the deployment plan based on your cluster’s actual capacity planning. Note that since the DataNode used some features of linux kernal, so that the kernel version of servers which used for deploy DataNode must be later than 3.10.

In order to speed up read and write of meta data, the meta data is stored in memory, while the DataNode mainly occupies disk resources. To maximize the use of node resources, you can mix-deploy DataNode and MetaNode on the same node.

Role	Spec	Test	Product
Master	CPU	>=4C	>=8C
	Memory	>=4G	>=16G
	Kernel	>=3.10	>=3.10
	Nodes	3	3
DataNode	CPU	>=4C	>=4C
	Memory	>=4G	>=8G
	Kernel	>=3.10	>=3.10
	Disk Capacity	>=1TB	>=2TB
	Disk Type	sata | ssd	sata | ssd
	File System	xfs | etx4	xfs | etx4
	Nodes	>=3	100~1000
MetaNode	CPU	>=4C	>=8C
	Memory	>=8G	>=16G
	Kernel	>=3.10	>=3.10
	Nodes	>=4	100~1000
Client	CPU	>=2C	>=2C
	Memory	>=4G	>=1G
	Kernel	>=3.10	>=3.10

Capacity Planning

First of all, you have to assess the highest expected number of files and storage capacity of the cluster in the future. Secondly, you need to know the machine resources you currently have, and the total memory, CPU cores, and disks on each machine. If you have been clear about those statistics, you can use the empirical reference values ​​given in the second section to see which scale your current environment belongs to, what file size it can carry,or you need to prepare for the current file experience requirements How many resources to prevent frequent expansion of machine resources.

Total File Count	Total File Size	Total memory	Total Disk Space
1,000,000,000	10PB	2048 GB	10PB

The higher the proportion of large files, the greater the MetaNode pressure.

Of course, if you feel that the current resources are adequately used, you don’t need to meet the capacity growth requirements all at once. Then you can pay attention to the capacity warning information of MetaNode/DataNode in time. When the memory or disk is about to run out, dynamically increase MetaNode/DataNode to adjust the capacity. In other words, if you find that the disk space is not enough, you can increase the disk or increase DataNode. If you find that all MetaNode memory is too full, you can increase MetaNode to relieve memory pressure.

Multi-Zone Deploy

If you want the cluster to support fault tolerance in the computer room, you can deploy a ChubaoFS cluster across computer rooms. At the same time, it should be noted that since the communication delay between computer rooms is higher than that of a single computer room, if the requirements for high availability are greater than low latency, you can choose a cross-computer room deployment solution. If you have higher performance requirements, it is recommended to deploy clusters in a single computer room. Configuration scheme: Modify the zoneName parameter in the DataNode/MetaNode configuration file, specify the name of the computer room where you are, and then start the DataNode/MetaNode process, the computer room will be stored and recorded by the Master along with the registration of DataNode/MetaNode.

Create a single zone volume:

$ cfs-cli volume create {name} --zone-name={zone}

In order to prevent volume initialization failure in a single computer room, please ensure that the DataNode of a single computer room is not less than 3 and MetaNode is not less than 4.

Create a cross-zone volume:

$ cfs-cli volume create {name} --cross-zone=true

Q&A

• If you are new to ChubaoFS and want to start quickly, please refer to Run Cluster on Docker
• If you are interested in ChubaoFS and want to conduct a performance test before applying it in the product environment, please refer to Performance
• If you have completed the assessment of ChubaoFS and want to put it into product environment, and want to learn how to carry out capacity planning and environment preparing, please refer to Environment Requirements and Capacity Planning
• If you want to know about best practices of ChubaoFS, please refer to Use Cases
• If you encounter some problems in the product environment, the following content may help to solve your problems.

For the convenience of description, we define the following keyword abbreviations

Full Name	Abbreviation
Data Partition	dp
Meta Partition	mp
Data Partition Replica	dpr
Meta Partition Replica	mpr
NodeSet	ns
DataNode	dn
MetaNode	mn

Compile

1. Compile ChubaoFS on one machine, but it cannot be started when deployed to other machines.

First please make sure to use the PORTABLE=1 make static_lib command to compile RocksDB, then use the ldd command to check whether the dependent libraries are installed on the machine. After installing the missing libraries, execute the ldconfig command.

2. A problem similar to undefined reference to ‘ZSTD_versionNumber’.

There are two solutions.

• It can be compiled by adding the specified library to CGO_LDFLAGS.

For example: CGO_LDFLAGS="-L/usr/local/lib -lrocksdb -lzstd". This requires that the zstd library is also installed on other deployment machines.

• Remove the script that automatically detects whether the ‘zstd’ library is installed.

Example of file location: rockdb-5.9.2/build_tools/build_detect_platform

The deleted content is as follows.

# Test whether zstd library is installed
    $CXX $CFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
    #include <zstd.h>
    int main() {}
EOF
    if [ "$?" = 0 ]; then
        COMMON_FLAGS="$COMMON_FLAGS -DZSTD"
        PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lzstd"
        JAVA_LDFLAGS="$JAVA_LDFLAGS -lzstd"
    fi

Node & Disk Failure

If a node or disk fails, you can take the failed node or disk offline by decommission command.

1. Decommission Datanode/Metanode

   $ cfs-cli metanode/datanode decommission 192.168.0.21:17210
   

   • At what time to decommission mn/dn？

   If a node fails and cannot reboot, it has to be removed from the cluster and the partitions on this machine is automatically migrated to other healthy nodes.

   • The decommission caused an acute increasing on disk io and network io.

   The decommission will trigger automatic partition migration witch consumes a lot of network resources. Therefore, try to execute the decommission during off peak hours and avoid taking multiple nodes decommission at the same time.

   • What is the sign of decommission completion？

     $ cfs-cli datapartition check
     

     If bad partition ids is empty, the decommission is done.

     • Common error1: There is no mn available, all mn memory or disks are full at this time, new mn needs to be added to the cluster
     • Common error2: The port number is wrong, the identifier of each mn should be a combination of ip+port , and port is the listen port in the mn configuration file, which cannot be wrong

2. Decommission Disk

If a disk fails but the node is healthy, you can partially decommission the disk. Similarly, as with dn/mn decommission, please try to avoid multiple decommission operations during off peak hours.

$ cfs-cli disk decommission {disk} 192.168.0.11:17310

If correct, all the dp on the disk is migrated to other disks or nodes.

The common error is similar to decommission dn.

3. Decommission Data Partition/Meta Partition

   $ cfs-cli datapartition decommission 192.168.0.11:17310 {Partition ID}
   

If correct, the dp is migrated to other nodes.

• At what time to decommission a partition？

  • There is too many partitions on one node, decommission some partitions to reduce the pressure on this node.
  • Take a small step instead of dn/mn decommission to prevent overloading of the cluster.

• The common error is similar to decommission dn/mn.

4. If the disk is full, will it explode?

It is recommended set reserved space in the dn startup json file, it is behind the disk path parameter, disk: “{disk path}:{reserved space}”. If the remaining space is less than reserved space, the dn turns readonly.

Performance of Data and Metadata

1. How does ChubaoFS compare with its alternatives ?

• Ceph

Ceph (https://ceph.com/) is a widely-used free-software storage platform. It can be configured for object storage, block storage, as well as file system. But many nice features provided by Ceph (e.g., various storage backends) also make it very complicated and difficult to learn and deploy. In addition, in certain scenarios such as metadata operations and small file operations, its performance in a multi-client environment can be hard to optimize.

• GFS and HDFS

GFS and its open source implementation HDFS (https://github.com/apache/hadoop) are designed for storing large files with sequential access. Both of them adopt the master-slave architecture, where the single master stores all the file metadata. Unlike GFS and HDFS, ChubaoFS employs a separate metadata subsystem to provide a scalable solution for the metadata storage so that the resource manager has less chance to become the bottleneck.

• Hadoop Ozone

Hadoop Ozone is a scalable distributed object storage system designed for Hadoop. It was originally proposed to solve the problem of HDFS namenode expansion. It reconstructs the namenode metadata management part of hdfs and reuses the datanode of hdfs. ChubaoFS has many of the same design concepts like ozone such as: supporting for volume isolation, compatible with both raft/master-slave synchronous replication mechanisms, implenting for s3-compatible interfaces. In addition, ChubaoFS’s POSIX fuse-client interface supports random file reading and writing, and optimizes reading and writing of small files.

• Haystack

Haystack from Facebook takes after log-structured filesystems to serve long tail of requests seen by sharing photos in a large social network. The key insight is to avoid disk operations when accessing metadata. ChubaoFS adopts similar ideas by putting the file metadata into the main memory.

However, different from Haystack, the actually physical offsets instead of logical indices of the file contents are stored in the memory, and deleting a file is achieved by the punch hole interface provided by the underlying file system instead of relying on the garbage collector to perform merging and compacting regularly for more efficient disk utilization. In addition, Haystack does not guarantee the strong consistency among the replicas when deleting the files, and it needs to perform merging and compacting regularly for more efficient disk utilization, which could be a performance killer. ChubaoFS takes a different design principle to separate the storage of file metadata and contents. In this way, we can have more flexible and cost-effective deployments of meta and data nodes.

• Public Cloud

Windows Azure Storage (https://azure.microsoft.com/en-us/) is a cloud storage system that provides strong consistency and multi-tenancy to the clients. Different from ChubaoFS, it builds an extra partition layer to handle random writes before streaming data into the lower level. AWS EFS (https://aws.amazon.com/efs/) is a cloud storage service that provides scalable and elastic file storage. Depending on the use cases, there could be a considerable amount of cost associated with using these cloud storage services.

• Others

GlusterFS (https://www.gluster.org/) is a scalable distributed file system that aggregates disk storage resources from multiple servers into a single global namespace. MooseFS (https://moosefs.com/) is a fault- tolerant, highly available, POSIX-compliant, and scalable distributed file system. However, similar to HDFS, it employs a single master to manage the file metadata.

2. If the scale of metadata is huge, how to improve cluster performance?

The metadata of ChubaoFS is stored in the memory. Expanding memory of the mn or expanding the amount of mn horizontally will significantly improve the metadata performance and support a large number of small files.

3. If a dn/mn is added to the cluster, will it be automatically rebalanced, for example the dp/mp on the old node are migrated to the new node?

No. Considering that the rebalance may cause overloading and risks data loss, it will not automatically rebalance. If you want the new node to carry more dp/mp to disperse the pressure of the old node, you can create new dp for this volume and the new dp may locate on new nodes, or you can decommission the dp on the old node.

4. There are scenes of batch delete files witch cause cluster overloads

You can set and view the background file deletion rate by following command, the default value is 0, which means unlimited. It is recommended to set markdeleterate=1000, and then dynamically adjust it according to the cpu status of the nodes in the cluster.

$ cfs-cli cluster info
$ cfs-cli cluster delelerate -h
 Set delete parameters

 Usage:
   cfs-cli cluster delelerate [flags]

 Flags:
       --auto-repair-rate string         DataNode auto repair rate
       --delete-batch-count string       MetaNode delete batch count
       --delete-worker-sleep-ms string   MetaNode delete worker sleep time with millisecond. if 0 for no sleep
   -h, --help                            help for delelerate
       --mark-delete-rate string         DataNode batch mark delete limit rate. if 0 for no infinity limit

Capacity Management

1. What if the capacity of Volume is used-out?

$ cfs-cli volume expand {volume name} {capacity / GB}

2. How to optimize the read/write performance from the Volume side？

The more dp that can be read and written, the better read and write performance of the Volume. ChubaoFS adopts a dynamic space allocation mechanism. After creating a Volume, it will pre-allocate a certain data partition dp for the Volume. When the number of dp that can be read and written is less than 10, the dp number will be automatically expanded by a step of 10. If you want to manually increase the number of readable and writable dp, you can use the following command:

$ cfs-cli volume create-dp {volume name} {number}

The default size of a dp is 120GB. Please add dp based on the actual cost to avoid overdrawing all dp.

3. How to reclaim the excess space of Volume

$ cfs-cli volume shrink {volume name} {capacity / GB}

If the set value is less than %120 of the used amount, the operation fails.

5. What if the cluster space is not enough?

Prepare the new dn and mn, start it by json configuration file with the current master hosts, and the dn/mn will be automatically added to the cluster.

Zone

Setting the zone can prevent the failure of a single zone witch causes the entire cluster unavailable. If parameter zoneName is set correctly, the node will automatically join the zone(cell).

1. See zones list

$ cfs-cli zone list

2. What if accidentally set the volume zone by mistake?

$ cfs-cli volume update {volume name}  --zone-name={zone name}

2. What happens if MetaNde and DataNode do not set zone?

Most of the parameters in the cluster have default values, and the default zone name is default. Please note that there must be enough dn and mn in a zone at the same time, otherwise, creating a volume in the zone will either fail to initialize data partitions or initialize meta partitions.

3. The meaning of NodeSet？

Each zone will have several NodeSets, and each NodeSet can carry 18 nodes by default. Because ChubaoFS implements multi-raft, each node starts a raft server process, and each raft server manages m raft instances on the node. If the other replication group members of these raft members are distributed on N nodes, then the raft heartbeat will be transmitted between N nodes. As the cluster scale expands, N will become larger. Through the NodeSet restriction, the heartbeat is relatively independent within the NodeSet, which avoids the heartbeat storm of the cluster dimension. We use the multi raft and nodeset mechanisms together to avoid the problem of the raft heartbeat storm.

4. How to distribute dp/mp of a volume in NodeSet?

dp/mp is evenly distributed in NodeSet. On dp/mp creating, it will locate on the NodeSet behind the NodeSet of last dp/mp overall.

5. How to plan the amount of NodeSet？

For 3-replicas dp/mp, a dp/mp will select the ns only when there are at least 3 available nodes in ns. count (ns) >= 18 * n + 3

Node status is abnormal

View node status through cli tool

$ cfs-cli datanode list
[Data nodes]
ID        ADDRESS               WRITABLE    STATUS
7         192.168.0.31:17310    No          Inactive
8         192.168.0.32:17310    No          Inactive
9         192.168.0.33:17310    Yes         Active
10        192.168.0.35:17310    Yes         Active
11        192.168.0.34:17310    Yes         Active

$ cfs-cli metanode list
[Meta nodes]
ID        ADDRESS               WRITABLE    STATUS
2        192.168.0.21:17210    No          Inactive
3        192.168.0.22:17210    No          Inactive
4        192.168.0.23:17210    Yes         Active
5        192.168.0.25:17210    Yes         Active
6        192.168.0.24:17210    Yes         Active

1. Reasons for Datanode WRITABLE=No

• The node is waiting for the decommission to complete
• Node disk is out of use
• The node has just started and is restoring data locally

2. Reasons for Metanode WRITABLE=No

• The node is waiting for the decommission to complete
• The node memory has reached totalMemory
• The node has just started and is restoring data locally

3. If one of the three masters in the cluster is broken, can the remaining two restarts provide services normally?

Yes. Since the Master uses the RAFT, it can provide services normally when the number of remaining nodes exceeds 50% of the total number of nodes.

4. Reasons for STATUS=Inactive

• The network connection between the node and the master is interrupted. You need to check the network status and restore the network connection
• The node process hangs, you need to check whether the server process of the node is abnormally terminated, at this time restart the process to recover

Upgrade

1. Steps

1. Download and unzip the latest binary file compression package from ChubaoFS official website https://github.com/chubaofs/chubaofs/releases
2. Freeze the cluster

$ cfs-cli cluster freeze true

3. Confirm the startup configuration file, do not change important information such as the data directory and port in the configuration file
4. Stop the old server process
5. Start the new server process
6. Check that the node status recovered healthy after the upgrade, IsActive: Active

$ cfs-cli datanode info 192.168.0.33:17310
[Data node info]
 ID                  : 9
 Address             : 192.168.0.33:17310
 Carry               : 0.06612836801123345
 Used ratio          : 0.0034684352702178426
 Used                : 96 GB
 Available           : 27 TB
 Total               : 27 TB
 Zone                : default
 IsActive            : Active
 Report time         : 2020-07-27 10:23:20
 Partition count     : 16
 Bad disks           : []
 Persist partitions  : [2 3 5 7 8 10 11 12 13 14 15 16 17 18 19 20]

7. Upgrade the next node (in order to reduce the impact on the client, especially in a relatively large user volume, you need to upgrade the MetaNode nodes one by one), the upgrade sequence is shown bellow

2. After upgrading a master, it is found that the monitoring system is not displayed in time?

Check whether the configuration of this master node is correct, especially the id; check the master error log to find out whether there is a large number of no leader errors, and query the keyword leaderChange in the master warn log to check the reason for the leader change, and then check the raft warn log for further analysis.

3. Can I modify the port number of the configuration during the upgrade?

No, ip+port constitutes the unique identifier of mn and dn instances, and will be treated as a new node after modification.

Update Configuration Online

1. Update mn threshold

$ cfs-cli cluster set threshold { value }

2. Update cluster configuration
3. Update Volume configuration

$ cfs-cli volume set -h
Set configuration of the volume
Usage:
  cfs-cli volume set [VOLUME NAME] [flags]
Flags:
      --authenticate string    Enable authenticate
      --capacity uint          Specify volume capacity [Unit: GB]
      --enable-token string    ReadOnly/ReadWrite token validation for fuse client
      --follower-read string   Enable read form replica follower
  -h, --help                   help for set
      --replicas int           Specify volume replicas number
  -y, --yes                    Answer yes for all questions
      --zonename string        Specify volume zone name

4. Update log level

An interface to update the log level of master, MetaNode, and DataNode online is provided:

$ http://127.0.0.1:{profPort}/loglevel/set?level={log-level}

Supported log-level: debug,info,warn,error,critical,read,write,fatal

Update Configuration Offline

1. Update master IP

After the ip address of the three-node master is replaced, all mn, dn and other applications that reference the master ip address need to be restarted after modifying the configuration.

2. Update DataNode/MetaNode port number

It is not recommended to modify the dn/mn port. Because dn/mn is registered through ip:port in the master. If the port is modified, the master will consider it as a brand new node, and the old node is in the state of Inactive.

3. Update MetaNode totalMemory

totalMemory refers to the total memory size of MetaNode. When the memory usage of MetaNode is higher than this value, MetaNode becomes read-only. Usually this value is smaller than the node memory. If MetaNode and DataNode are deployed in a mixed deployment, extra memory needs to be reserved for the DataNode.

4. Update DataNode reservedSpace

In the dn startup configuration json file, the number in the second half of the disk parameter is the value of Reserved Space, unit(byte).

{ ...
"disks": [
   "/cfs/disk:10737418240"
  ],
  ...
  }

5. For more configurations, see Resource Manager (Master) Data Subsystem Meta SubsystemClient.

Handle Logs

1. What to do if tens of GB of logs are generated every day, which takes up too much disk space?

In a production environment, you can set the log level to warn or error, which will significantly reduce the amount of logs.

$ http://127.0.0.1:{profPort}/loglevel/set?level={log-level}

Supported log-level: debug,info,warn,error,critical,read,write,fatal

2. Datanode warn log

checkFileCrcTaskErr clusterID[xxx] partitionID:xxx File:xxx badCrc On xxx:

Analysis: The Master schedules the dn to check the crc data every few hours. This error indicates that the crc check failed and the file data is wrong. At this time, it is necessary to further analyze the error according to the partitionID and File in the warn message and with the assistance of the info log.

3. Datanode error log
4. Master error log

clusterID[xxx] addr[xxx]_op[xx] has no response util time out

Analysis:The response timed out when the Master sends the [Op] command to mn or dn, check the network between Master and mn/dn; check whether the dn/mn service process is alive.

5. Master warn log
6. Metanode error log

Error metaPartition(xx) changeLeader to (xx):

Analysis:Leader change, a normal action.

inode count is not equal, vol[xxx], mpID[xx]

Analysis:The number of inode is inconsistent. Because as long as two of the three copies are successful, the write is successful, so there will be inconsistencies in the three copies. Check the log for the specific reason.

7. Metanode warn log
8. Client warn log

operation.go:189: dcreate: packet(ReqID(151)Op(OpMetaCreateDentry)PartitionID(0)ResultCode(ExistErr)) mp(PartitionID(1) Start(0) End(16777216) Members([192.168.0.23:17210 192.168.0.24:17210 192.168.0.21:17210]) LeaderAddr(192.168.0.23:17210) Status(2)) req({ltptest 1 1 16777218 test.log 420}) result(ExistErr)

Analysis: ExistErr indicates that the file name already exists during the rename operation. It is an upper-level business operation problem. Maintainers can ignore this.

extent_handler.go:498: allocateExtent: failed to create extent, eh(ExtentHandler{ID(xxx)Inode(xxx)FileOffset(xxxx)StoreMode(1)}) err(createExtent: ResultCode NOK, packet(ReqID(xxxxx)Op(OpCreateExtent)Inode(0)FileOffset(0)Size(86)PartitionID(xxxxx)ExtentID(xx)ExtentOffset(0)CRC(0)ResultCode(IntraGroupNetErr)) datapartionHosts(1.1.0.0:17310) ResultCode(IntraGroupNetErr))

Analysis: The client sends a request to create an extent to an mp and fails, it will soon try to request another mp.

9. Client error log

appendExtentKey: packet(%v) mp(%v) req(%v) result(NotExistErr)

Analysis: This error indicates that the file was deleted before written, which is an upper-level business operation problem. Maintainers can ignore this.

conn.go:103:sendToMetaPartition: retry failed req(ReqID(xxxx)Op(OpMetaInodeGet)PartitionID(0)ResultCode(Unknown ResultCode(0)))mp(PartitionID(xxxx) Start(xxx) End(xxx) Members([xxx xxxx xxxx]) LeaderAddr(xxxx) Status(2)) mc(partitionID(xxxx) addr(xxx)) err([conn.go 129] Failed to read from conn, req(ReqID(xxxx)Op(OpMetaInodeGet)PartitionID(0)ResultCode(Unknown ResultCode(0))) :: read tcp 10.196.0.10:42852->11.196.1.11:9021: i/o timeout) resp(<nil>)

Analysis 1:The network connection between the client and the MetaNode is abnormal. According to the error message “10.196.0.10:42852->11.196.1.11:9021”, check whether the network between the two addresses is normal

Analysis 2:Check if the MetaNode process hangs on “11.196.1.11:9021”

10. Raft warn log
11. Raft error log

raft.go:446: [ERROR] raft partitionID[1105] replicaID[6] not active peer["nodeID":"6","peerID":"0","priority":"0","type":"PeerNormal"]

Analysis: This is caused by excessive network pressure and increased delay. After the raft election interval is exceeded, the raft replication group loses the leader. After the network is restored, re-elect the leader, the error will disappear by itself.

Data Loss and Consistence

1. All data of a single dn/mn is lost

This situation can be equivalent to a dn/mn failure. You can log off the node through decommission, and then restart the node to re-register the node to the Master, and the Master will treat it as a new member.

2. Accidentally deleted file data in a dp directory in dn

dn has the function of automatically repairing data. If the data has not been repaired for a long time, you can manually restart the current dn process, which will trigger the data repair process.

Fuse Client

1. Memory and performance optimization issues

• The Fuse client occupies too much memory, which has a large impact on other services

  Offline: Set the readRate and writeRate parameters in the configuration file and restart the client.

  Online: http://{clientIP}:{profPort} /rate/set?write=800&read=800

• For more, see(https://chubaofs.readthedocs.io/zh_CN/latest/user-guide/fuse.html)

2. Mount issues

• Does it support subdirectory mounting?

Yes. Set subdir in the configuration file

• What are the reasons for the mount failure

  • If you see the following output,

  $ ... err(readFromProcess: sub-process: fusermount: exec: "fusermount": executable file not found in $PATH)
  

  Check if fuse is installed, if not, install it

  $ rpm –qa|grep fuse
    yum install fuse
  

  • Check if the mount directory exists
  • Check whether the mount point directory is empty
  • Check whether the mount point has been umount
  • Check whether the mount point status is normal. If the following message appears on the mount point mnt, you need to umount first, and then start the client

  $ ls -lih
  ls: cannot access 'mnt': Transport endpoint is not connected
  total 0
  6443448706 drwxr-xr-x 2 root root 73 Jul 29 06:19 bin
   811671493 drwxr-xr-x 2 root root 43 Jul 29 06:19 conf
  6444590114 drwxr-xr-x 3 root root 28 Jul 29 06:20 log
           ? d????????? ? ?    ?     ?            ? mnt
   540443904 drwxr-xr-x 2 root root 45 Jul 29 06:19 script
  

  • Check whether the configuration file is correct, master address, volume name and other information
  • If none of the above problems exist, locate the error through the client error log to see if it is the mount failure caused by the MetaNode or master service

3. IO issues

• IOPS is too high and the client’s memory usage exceeds 3GB or even higher. Is there any way to limit IOPS?

Limit the frequency of client response to io requests by modifying the client rate limit.

#see current iops:
$ http://[ClientIP]:[profPort]/rate/get
#set iops, default:-1(no limits)
$ http://[ClientIP]:[profPort]/rate/set?write=800&read=800

• io delay too high for ls or other operations

  • Because the client reads and writes files through the http protocol, please check whether the network status is healthy
  • Check whether there is an overloaded mn, whether the mn process is hanging, you can restart mn, or expand a new mn to the cluster and take the mp on the overloaded mn decommission to relieve the pressure of mn

4. Does ChubaoFS provide strong consistence guarantees?

No.ChubaoFS has relaxed POSIX consistency semantics, i.e., instead of providing strong consistency guarantees, it only ensures sequential consistency for file/directory operations, and does not have any leasing mechanism to prevent multiple clients writing to the same file/directory. It depends on the upperlevel application to maintain a more restrict consistency level if necessary.

5. Is it feasible to kill the client to directly stop the client service

No. It is recommended to umount first. After umount, the client process will automatically stop.