Difference between revisions of "Filesystems:Frontenac"

From CAC Wiki
Jump to: navigation, search
 
(30 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
The Frontenac cluster uses a shared GPFS filesystem for all file storage. User files are located under <code>/global/home</code>, shared project space under <code>/global/project</code>, and network scratch space under <code>/global/scratch</code>. In addition to the network storage, each compute node has a 1.5TB local hard disk for fast access to local scratch space by jobs using the location specified by the <code>$TMPDISK</code> environment variable. All files in the local scratch space are assumed to be deleted automatically when corresponding jobs finish.
+
 
 +
'''Important Note: Due to the transition to cost-recovery service many of the details of file system organization on the Frontenac GPFS file system are subject to change. Please refer back to these pages occasionally to stay abreast of these changes.'''
 +
 
 +
==Overview== <!--T:1-->
 +
 
 +
<!--T:2-->
 +
The Frontenac cluster uses a shared [https://www.ibm.com/support/knowledgecenter/en/SSFKCN/gpfs_welcome.html GPFS filesystem] for all file storage.  
 +
User files are located under <code>/global/home</code> of '''500GB''' quota, shared project space under <code>/global/project</code>,  
 +
and network scratch space under <code>/global/scratch</code> of '''5TB''' quota.  
 +
In addition to the network storage, compute nodes have between up to '''1.5TB''' local hard disk for fast access to local scratch space by jobs using  
 +
the location specified by the <code>$TMPDISK</code> environment variable. All files in the local scratch space are assumed to be deleted automatically  
 +
when corresponding jobs finish.
 +
<!--T:3-->
 +
Note that it is the user's responsibility to manage the age of their data: these file systems do not provide archiving.
 +
If data are no longer needed, they need to be moved off the system. If you need assistance with this, please contact us.
 +
This is especially important as we are charging for storage in the Terabyte range. At present, ''nearline'' data (see explanation below) are free, but ''project'' data (see below) are subject to charges on an annual basis. Detail about our cost structure can be found at [[Frontenac:Fees|our Fees Information Page]].
 +
 
 +
== Storage Areas == <!--T:5-->
 +
 
 +
Unlike your personal computer, our system has several storage spaces or file systems and you should ensure that you are using the right space for the right task. In this section we will discuss the principal file systems available, and the intended use of each one along with its characteristics. Storage options are distinguished by the available hardware, access mode and write system. Typically, most Compute Canada systems offer the following storage types:
 +
 
 +
<!--T:6-->
 +
;Global Parallel File System (GPFS)
 +
: This file system is visible on both login and compute nodes. Combining multiple disk arrays and fast servers, it offers excellent performance for large files and large input/output operations. Two types of storage are distinguished on such systems: long term storage and temporary storage (scratch). Performance is subject to variations caused by other users.
 +
;Local Filesystem
 +
: This is a local hard drive attached to each of the nodes. Its advantage is high performance (because it is rarely shared). Its disadvantage is that local files must be re-copied to a global area to be vi sible on other nodes such as the login ()workup) node. Typically, local disk is regularly "cleaned", i.e. data kept there are considered transitory.
 +
;RAM (memory) Filesystem
 +
: This is a filesystem that exists within a node's RAM, so it reduces the available memory. This makes it very fast but low-capacity. A RAM disk must be cleaned at the end of a job.
 +
 
 +
<!--T:7-->
 +
The following table summarizes the properties of these storage types.
 +
 
 +
<!--T:8-->
 +
{| class="wikitable" style="font-size: 95%; text-align: center;"
 +
|+align="bottom" style="color:#e76700;"|''Description of storage type''
 +
! scope="col" width="120px" | Type
 +
! scope="col" width="120px" | Accessibility
 +
! scope="col" width="120px" | Throughput
 +
! scope="col" width="120px" | Latency
 +
! scope="col" width="120px" | Longevity
 +
|-
 +
|GPFS (/global/home, /global/project ...)
 +
|style="background-color:#00c000;"|All nodes
 +
|style="background-color:#ffff00;"|Fair
 +
|style="background-color:#ff0000;"|High
 +
|style="background-color:#00c000;"|Long term
 +
|-
 +
|GPFS (/global/scratch)
 +
|style="background-color:#00c000;"|All nodes
 +
|style="background-color:#ffff00;"|Fair
 +
|style="background-color:#ff0000;"|High
 +
|style="background-color:#ffff00;"|Short term (periodically cleaned)
 +
|-
 +
|Local Filesystem (TMPDIR)
 +
|style="background-color:#ff0000;"|Local to the node
 +
|style="background-color:#ffff00;"|Fair
 +
|style="background-color:#ffff00;"|Medium
 +
|style="background-color:#ff0000;"|Very short term
 +
|-
 +
|Memory (RAM) FS
 +
|style="background-color:#ff0000;"|Local to the node
 +
|style="background-color:#00c000;"|Good
 +
|style="background-color:#00c000;"|Very low
 +
|style="background-color:#ff0000;"|Very short term, cleaned after every job
 +
|}
 +
'''Throughput''' describes the efficiency of the file system for large operations. Sometimes also called "bandwidth" in the context of FS-IO.
 +
 
 +
'''Latency''' describes the efficiency of the file system for small operations. Low latency is good.
 +
 
 +
== Quotas == <!--T:10-->
 +
 
 +
<!--T:11-->
 +
On our cluster, each user has access to the /global/home and /global/scratch spaces by default and each group has access to project space in /global/project. These areas are subject to disk quota
 +
<!--T:12-->
 +
{| class="wikitable" style="font-size: 95%; text-align: center;"
 +
|+Filesystem Characteristics
 +
! Area
 +
! Quota
 +
! Backup ?
 +
! Purge ?
 +
! Default ?
 +
! On Nodes?
 +
|-
 +
|/global/home
 +
| 500 GB
 +
| Yes
 +
| No
 +
| Yes
 +
| Yes
 +
|-
 +
| /global/scratch
 +
| 5 TB
 +
| No
 +
| Yes
 +
| Yes
 +
| Yes
 +
|-
 +
| /global/project
 +
| 
 +
| Yes
 +
| No
 +
| Yes
 +
| Yes
 +
 
 +
|}
 +
 
 +
== Some Tips == <!--T:9-->
 +
* Avoid text format files for large data.
 +
* Use local storage for temporary files. The scheduler provides this (<code>$TMPDIR or $TMPDISK</code>) which is created when you job starts on a compute node, e.g. TMPDISK=/lscratch/slurm-job-6363084.
 +
* Searches should be done in memory rather than on disk.
 +
* Regularly clean up data, especially in scratch.
 +
* Unused files that have to be kept should be moved off-system.

Latest revision as of 13:38, 27 September 2023

Important Note: Due to the transition to cost-recovery service many of the details of file system organization on the Frontenac GPFS file system are subject to change. Please refer back to these pages occasionally to stay abreast of these changes.

Overview

The Frontenac cluster uses a shared GPFS filesystem for all file storage. User files are located under /global/home of 500GB quota, shared project space under /global/project, and network scratch space under /global/scratch of 5TB quota. In addition to the network storage, compute nodes have between up to 1.5TB local hard disk for fast access to local scratch space by jobs using the location specified by the $TMPDISK environment variable. All files in the local scratch space are assumed to be deleted automatically when corresponding jobs finish. Note that it is the user's responsibility to manage the age of their data: these file systems do not provide archiving. If data are no longer needed, they need to be moved off the system. If you need assistance with this, please contact us. This is especially important as we are charging for storage in the Terabyte range. At present, nearline data (see explanation below) are free, but project data (see below) are subject to charges on an annual basis. Detail about our cost structure can be found at our Fees Information Page.

Storage Areas

Unlike your personal computer, our system has several storage spaces or file systems and you should ensure that you are using the right space for the right task. In this section we will discuss the principal file systems available, and the intended use of each one along with its characteristics. Storage options are distinguished by the available hardware, access mode and write system. Typically, most Compute Canada systems offer the following storage types:

Global Parallel File System (GPFS)
This file system is visible on both login and compute nodes. Combining multiple disk arrays and fast servers, it offers excellent performance for large files and large input/output operations. Two types of storage are distinguished on such systems: long term storage and temporary storage (scratch). Performance is subject to variations caused by other users.
Local Filesystem
This is a local hard drive attached to each of the nodes. Its advantage is high performance (because it is rarely shared). Its disadvantage is that local files must be re-copied to a global area to be vi sible on other nodes such as the login ()workup) node. Typically, local disk is regularly "cleaned", i.e. data kept there are considered transitory.
RAM (memory) Filesystem
This is a filesystem that exists within a node's RAM, so it reduces the available memory. This makes it very fast but low-capacity. A RAM disk must be cleaned at the end of a job.

The following table summarizes the properties of these storage types.

Description of storage type
Type Accessibility Throughput Latency Longevity
GPFS (/global/home, /global/project ...) All nodes Fair High Long term
GPFS (/global/scratch) All nodes Fair High Short term (periodically cleaned)
Local Filesystem (TMPDIR) Local to the node Fair Medium Very short term
Memory (RAM) FS Local to the node Good Very low Very short term, cleaned after every job

Throughput describes the efficiency of the file system for large operations. Sometimes also called "bandwidth" in the context of FS-IO.

Latency describes the efficiency of the file system for small operations. Low latency is good.

Quotas

On our cluster, each user has access to the /global/home and /global/scratch spaces by default and each group has access to project space in /global/project. These areas are subject to disk quota

Filesystem Characteristics
Area Quota Backup ? Purge ? Default ? On Nodes?
/global/home 500 GB Yes No Yes Yes
/global/scratch 5 TB No Yes Yes Yes
/global/project Yes No Yes Yes

Some Tips

  • Avoid text format files for large data.
  • Use local storage for temporary files. The scheduler provides this ($TMPDIR or $TMPDISK) which is created when you job starts on a compute node, e.g. TMPDISK=/lscratch/slurm-job-6363084.
  • Searches should be done in memory rather than on disk.
  • Regularly clean up data, especially in scratch.
  • Unused files that have to be kept should be moved off-system.
Retrieved from ‘https://info.cac.queensu.ca/wiki/index.php?title=Filesystems:Frontenac&oldid=2637’