Frontenac:Migration

Migrating to the new Frontenac (CAC) cluster

This is a basic guide for users of our current CentOS 6 production systems ("SW cluster") to explain and facilitate migration to our new CentOS 7 systems ("Frontenac", "CAC cluster").

Note: We are in the final phase of the migration process. All users will gain access to the new systems by mid-November, and lose acess to the old systems in early January 2018. Scheduling of new jobs on the old system will stop in mid-December! Please make yourself familiar with the new systems.

Why migrate ?

Our systems underwent a substantial refresh last year with the retirement of the Solaris-based M9000 systems, and their replacement by new X86/Intel based hardware. This hardware was largely added to the existing "SW cluster" and eventually replaced it completely. However, this gradual replacement did not address issues in the base structure of that cluster, such as an old scheduler system, and a less than cutting-edge file system. To enable our users to make efficient use of the new hardware, we decided that it is time for a re-design of our main compute cluster. Some of our storage components reach their "end of life" phase and will be retired within a year.

Rather than permanently operating two separate clusters, we will move both our users and the compute hardware from one cluster/network to the other. In the interest of consistency, we can not make this process optional. We must move all our users to the new cluster by early 2018 when service contracts for the old components run out.

What's Different ?

The new cluster is based on a newer version of the same CentOS operating system. We have replaced the scheduler with SLURM, which is the same as is used on the new Compute Canada "GP" systems. We also replaced the "use" system by the more powerful and standard "lmod". Here are the main changes in table format.

	new SW (Linux) cluster	new CAC (Frontenac) cluster
Operating system	CentOS 6	CentOS 7
File system type	ZFS	GPFS
Scheduler	Sun Grid Engine (SGE)	SLURM
Software manager	usepackage	lmod
Backup management	samfs	Hierarchical Storage Management (HSM)

Migration Time Table

Different users will migrate at different times. We have been moving data to the new file system for months, so that at the time when "it's your turn" your data will already be available on the new system. Here is a month-by-month outline of who will move when. If you want to migrate ahead of schedule, or you have compelling reasons to delay the move, please get in touch with us at cac.help@queensu.ca

Month (2017)	Who moves ?
September	De-actived users User who have not run a scheduled job for > 6 months Volunteers
October	New accounts (i.e. new users will be going straight to Frontenac) User who have not run a scheduled job for > 3 months Volunteers
November	New accounts (i.e. new users will be going straight to Frontenac) Everyone (middle of the month)
December	New accounts (i.e. new users will be going straight to Frontenac) Everyone

We will transfer hardware from the "old" cluster (SW) to the new one (Frontenac) to accommodate the migrated users. This means that in the transition period, the old cluster will gradually become smaller while the new one grows. Dedicated hardware will be moved when its users migrate.

IMPORTANT DEADLINES

In the final phase of the migration process, all users receive a notification email and are asked to make themselves familiar wit the new systems. Here is a list of important dates that our users should keep in mind when planning to use our systems in the time period between November 2017 and February 2018.

Date	Migration Event	System
November 6, 2017	Scheduling halted for all nodes with more than 24 cores	SW ("old system")
November 15, 2017	User notification by email All users receive access to new systems	Frontenac ("new system")
December 15, 2017	Data synchronization stops User data that differ on the two systems must be transferred by users Grid Engine Scheduler does not accept new submissions (scheduling stops, nodes "draining")	SW ("old system")
January 3, 2018	All running jobs are terminated Remaining hardware is transferred to new system	SW ("old system")	-	January 8, ,2018	User access to sflogin0/swlogin1 closed SNOlab (SX) cluster jobs terminated SNOlab (SX) login nodes closed	SW ("old system")

We will transfer hardware from the "old" cluster (SW) to the new one (Frontenac) to accommodate the migrated users. This means that in the transition period, the old cluster will gradually become smaller while the new one grows. Dedicated hardware will be moved when its users migrate.

Migration Schedule

The migration proceeds according to a scheme that was devised to minimize the impact on operations and user's research activities. Research groups migrate as a whole during a 1-month week time period. The migration procedure has three steps:

• 1 - Initiation of migration process
  • Email notification of the user (mid-November).
  • Create account on new cluster.
  • Issue temporary credentials to the new cluster and request initial login to change password.
• 2 - Rolling rsync of user data
  • Will be repeated until update requires less than 2 hrs
    • /home/hpcXXXX
    • /u1/work/hpcXXXX
    • /scratch/hpcXXXX if required
    • other directories if required
  • Users can access both new and old systems for 1 month.
    • Data on the old system that are newer than on the new one are rsync'ed.
• 3 - Final migration
  • Final rsync.
  • Jobs on old cluster are terminated.
  • User access to old system closed.

New cluster HowTo ...

• ... log into the system
• ... setup and use software
• ... find your way around the filesystems
• ... submit jobs using SLURM

Migration Q&A

• Q: Who migrates ?

A: Eventually, all of our users will migrate from the old SW cluster to the new "Frontenac" cluster

• Q: Can I use my old "stuff" ?

A: Much of the old data and software will be usable on the new systems. However, the data will have to be copied over as the new systems use a separate file system, and cross access is not possible.

• Q Do I have to re-compile ?

A: It is possible that you will have to re-compile some of the software you are using. We will assist you with this.

• Q: Do I copy my files over myself ?

A: No. We will do this for you in the course of the coming months.

• Q: Is this optional ?

A: Unfortunately not. We will move both user data and hardware according to a schedule.

• Q: Can I decide when to move ?

A: To a degree. We are open to "early adopters". Once they have been moved we move the bulk of users according to our schedule.

• Q: Will this disrupt my research ?

A: We will do our level best to keep disruptions to a minimum. We will give you a chance to "practise" on the new systems while you still have access to the old ones. Once you are on the new systems, access to the old ones will be cut to preserve data integrity.

Help

If you have questions that you can't resolve by checking documentation, email to cac.help@queensu.ca.