Flight Silo

This documentation is for an OpenFlight Labs Project which is in maintenance mode

Overview

Flight Silo allows users to connect to “silos” - cloud storage systems designed primarily to distribute software and projects, with more general file storage also available. Silo comes installed as part of the User Suite included with Flight Solo.

This section details the various commands used to manage silos, files and software.

Setting up platforms

Use flight silo type avail to list all available platform types and their current state. e.g.

[flight@chead1 ~]$ flight silo type avail
┌───────────┬───────────────────────────────┬──────────┐
│ Name      │ Description                   │ Prepared │
├───────────┼───────────────────────────────┼──────────┤
│ aws       │ Amazon Simple Storage Service │ true     │
│ openstack │ Openstack Swift Storage       │ true     │
└───────────┴───────────────────────────────┴──────────┘

Types are not prepared by default. To prepare a type you must become the root user, and then do flight silo type prepare <type>. e.g.

[root@chead1 ~]# flight silo type prepare aws
Preparing...
Type aws prepared for use

Types are the backend providers of storage, and each type can hold multiple silos. Silos are used to hold software and files. To find out more about these see the relevant page in this section, or use the command flight silo help to view available commands and command syntax.

Managing Silos

A silo is a storage space that is the same regardless of platform used. Even though a silo may appear as a directory in your cloud storage, it is highly recommended that a silo should only be managed by the command line tool.

OpenFlight Silo

The Flight Silo tool comes with the openflight silo, this is a read-only silo that provides some example files & software. So long as the AWS platform is prepared then the openflight repository can be interacted with to pull files and software.

Viewing Silos

To see the available silos on the system use the command flight silo repo list.

[flight@chead1 ~]$ flight silo repo list
┌────────────┬───────────────────────────────────┬──────────┬─────────┬────────────┐
│ Name       │ Description                       │ Platform │ Public? │ ID         │
├────────────┼───────────────────────────────────┼──────────┼─────────┼────────────┤
│ mysilo1    │                                   │ aws      │ false   │ ABCDE123   │
│ openflight │ Openflight software and resources │ aws      │ true    │ OPENFLIGHT │
└────────────┴───────────────────────────────────┴──────────┴─────────┴────────────┘

Silos that are defined locally and no longer exist upstream will be marked in yellow. This can occur when a silo has been deleted from a different system.

Creating and Adding Silos

You can add an already existing silo with the command flight silo repo add. All questions asked will be the same as for creation, except that the answers will be used to find an existing silo.

To create a silo use the command flight silo repo create. This will take you through a series of questions:

• Provider type - Which provider the silo should be created on.
• Silo name - What the name of the silo should be.

After this point, any further questions depend on the chosen platform.

AWS

• Region - The region the silo should be created in.
• Access key ID - The ID for a valid aws access key.
• Secret access key - The secret key for a valid aws access key.

More information about AWS access keys can be found in the AWS documentation.

OpenStack

• Endpoint URL - The URL of the OpenStack Swift Storage service to store this silo
• Access key ID - The ID for a valid access key generated by OpenStack EC2-like credentials
• Secret access key - The secret key for the above access key

Removing and Deleting Silos

If you no longer wish to have access to a silo on your machine, you can run the command flight silo repo remove <name>. This means that in order to access the silo again you would need to use the add command.

A silo can be deleted with flight silo repo delete <name>. Unlike the remove command, the silo could not be re-added later as it is fully deleted upstream along with all of its contents.

Setting a default silo

If a default silo has been set, then commands that require a silo to be specified in the argument will use the default instead. Set a default with the command flight silo set-default e.g.

[flight@chead1 ~]$ flight silo set-default openflight
Default silo set to: openflight

Editing Silo Information

When a silo is created it is named and the description is left blank. Both the name and description can be modified using the flight silo repo edit command. For example:

[flight@chead1 ~]$ flight silo repo edit mysilo1
Silo name: myfilesilo1
Silo description: This is my silo for sharing files between ephemeral cloud systems!
Updating silo details...
Silo details updated

Any other systems that have a silo added locally will detect upstream changes and will require flight silo repo refresh to be run to ensure names and descriptions are synchronised with the upstream silo.

Managing Files

Silo provides simple file management, this allows users to save and restore their files

file list <silo>:<directory>

List the files in the given silo and directory. If no silo is specified then the default will be used. If no directory is given then the top-level of the silo file storage will be shown.

Example 1

[flight@chead1 ~]$ flight silo file list openflight:/openfoam
cavity-example.sh  motorBike.tar.gz  OpenFOAM-v2212.tar.gz

file push <source> <silo>:<destination>

Upload a file to a silo.If no silo is specified then the default will be used.

• --recursive - Uploads a whole directory and all of its contents. You must specify a directory rather than a file.
• --make-parent - Create parent directories if they don’t exist.

Example 1

[flight@chead1 ~]$ flight silo file push example.sh
Local file '/home/flight/example.sh' copied to remote '/example.sh'

Example 2

[flight@chead1 ~]$ flight silo file push dir/subdir/example.sh --make-parent
Local file '/home/flight/example.sh' copied to remote '/dir/subdir/example.sh'

file pull <silo>:<destination> <local destination>

Pull a file or directory from a silo. If no silo is specified then the default will be used. If no local destination is specified then the current working directory will be used.

• --recursive - Pulls a whole directory and all of its contents.

Example 1

[flight@chead1 ~]$ flight silo file pull openflight:/kubernetes/pod-launch-test.yaml
Pulling 'openflight:/kubernetes/pod-launch-test.yaml' into '/home/flight'...
File(s) downloaded to /home/flight

Example 2

[flight@aztest1 ~]$ flight silo file pull --recursive openflight:/openfoam/ openfoam-download-example
Pulling 'openflight:/openfoam/' into 'openfoam-download-example'...
File(s) downloaded to openfoam-download-example

file delete <silo>:<destination>

Delete a file in a silo. If no silo is specified then the default will be used.

• --recursive - Deletes a whole directory and all of its contents. You must specify a directory rather than a file.

Example 1

[flight@chead1 ~]$ flight silo file delete example.sh
Deleting remote file 'example.sh'...
Deleted remote file 'example.sh'

Managing Software

Software in the context of Flight Silo is software binaries that can be compiled in advance, and then kept in a silo so that they can be accessed from any number of HPC environments without needing to be recompiled. This provides users a method to manage and share their own software.

This page describes several commands associated with software which are run in the format flight silo <command> --<option>.

software delete <name> <version>

Delete a software binary from the default silo.

• --repo <silo> - Instead of using the default silo, specify which one to delete from.

Example 1

[flight@chead1 ~]$ flight silo software delete exampl 0.0
Deleting software 'exampl' version '0.0'...
Deleted software 'exampl' version '0.0'.

software pull <name> <version>

Download and extract a software binary from the default silo.

• --repo <silo> - Instead of using the default silo, specify which one to pull from.
• --dir <path> - Install software within <path> instead of the standard location. The order in which the destination path is decided is based on the following (in priority order):
  1. The value of the environment variable flight_SILO_software_dir
  2. The --dir argument
  3. The value of software_dir in ~/.config/flight/silo/config.yml
  4. The value of software_dir in /opt/flight/opt/silo/config.yml (this path may differ if a non-standard installation location has been chosen or if the source code has been cloned elsewhere)
• --overwrite - Overwrite local software if it exists.

Example 1

[flight@chead1 ~]$ flight silo software pull exampl 0.0
Downloading software 'exampl' version '0.0'...
Extracting software to '/home/flight/.local/share/flight/silo/software'...
Extracted software 'exampl' version '0.0 to '/home/flight/.local/share/flight/silo/software'...

software push <file> <name> <version>

Upload a software binary to the default silo.

• --repo <silo> - Instead of using the default silo, specify which one to push to.
• --force - Overwrite existing software in the silo if it exists.

Example 1

[flight@chead1 ~]$ flight silo software push exampl.tar.gz exampl 0.0
Uploading software 'exampl' version '0.0'...
Uploaded software 'exampl' version '0.0'.

software search <name>

List the software binaries in the default silo. If no name is given, then all software will be displayed.

• --repo <silo> - Instead of using the default silo, specify which one to search on.

Example 1 - With Search Term (Shows Versions)

[flight@chead1 ~]$ flight silo software search exampl
┌────────┬─────────┬────────┐
│ Name   │ Version │ Size   │
├────────┼─────────┼────────┤
│ exampl │ 0.0     │ 500 MB │
└────────┴─────────┴────────┘

Example 2 - With No Search Term

[flight@chead1 ~]$ flight silo software search
Showing latest 5 versions...
┌────────┬─────────┐
│ Name   │ Version │
├────────┼─────────┤
│ exampl │ 0.0     │
└────────┴─────────┘

Managing Migrations

Migrations provide a method for replicating Silo software installations on new clusters. This supports quicker time-to-science by reducing the admin work required to create duplicate environments in ephemeral cloud situations.

Every software pull made by the user will be recorded in an “archive”. This will detail what software version was installed from what silos and to which directories it was installed.

migration view

Show the state of the current migration archives and any others available to the system.

• --archive <archive> - Specify the archive to view instead of the currently enabled one

Example 1

[flight@gateway1 ~]$ flight silo migration view

Archives:
┌──────────┬─────────┬───────────┐
│ Archive  │ Status  │ Host Silo │
├──────────┼─────────┼───────────┤
│ wooldbfe │ enabled │ Undefined │
└──────────┴─────────┴───────────┘

Enabled archive details:
┌──────────┬──────────┬─────────┬───────────────────────┬──────────┬────────────┐
│ Type     │ Name     │ Version │ Path                  │ Absolute │ Silo Name  │
├──────────┼──────────┼─────────┼───────────────────────┼──────────┼────────────┤
│ software │ OpenFOAM │ 22.12   │ ~/apps/OpenFOAM/22.12 │ false    │ openflight │
└──────────┴──────────┴─────────┴───────────────────────┴──────────┴────────────┘

Software install destinations within home directories are portable between different users as they are not marked as “absolute” paths so, when migrating, will be installed into the home directory of whichever user is applying the migration

migration apply

Apply a migration archive to the current system. This will attempt to install the software within the archive to the recorded destinations.

• --archive <archive> - Specify the archive ID to view instead of the currently enabled one
• --ignore-missing-item - Don’t fail the apply if a silo, software package or version of a software package within the archive are not present
• --overwrite - Overwrite software locally if it already exists at the installation location

Example 1

[flight@gateway1 ~]$ flight silo migration apply --archive qsaoxvxj
Validating Archive 'qsaoxvxj'...
Migration for archive 'qsaoxvxj' started...

Migrating software my-software 1.0.0...
'my-software' '1.0.0' successfully migrated


Migrating software my-software 2.0.0...
'my-software' '2.0.0' successfully migrated

Migration All Done √

migration continue

Resume monitoring of software pull actions.

Example 1

[flight@gateway1 ~]$ flight silo migration continue
Migration Monitoring Enabled √

migration pause

Pause monitoring of software pull actions. After running this command any software installed with Silo will not be added to the currently enabled archive.

Example 1

[flight@gateway1 ~]$ flight silo migration pause
Migration Monitoring Disabled √

migration pull <archive>

Retrieve updated archive details from a silo.

Example 1

[flight@gateway1 ~]$ flight silo migration pull mysilo1
Obtaining silo migration archives...
Done √

migration push

Update remote silos with latest local archive information. Archives are pushed to their “Host Silo”, those without “Host Silos” are uploaded to the default silo.

• --repo <silo> - Specify a silo other than the default to store the local-only archives in

Example 1

[flight@gateway1 ~]$ flight silo migration push --repo mysilo1
Updating migration archives for Silo mysilo1...
All Done √

migration remove software <name> <version>

Remove a software item from the current archive.

• --archive <archive> - Remove from the specified archive instead of the current one.
• --all - Remove the software version from all archives

Example 1

[flight@gateway1 ~]$ flight silo migration remove software OpenFOAM 22.12
Software 'OpenFOAM 22.12' migration record has been removed from archive 'wooldbfe'

migration switch

Change from the currently enabled archive to a new, empty archive.

• --archive <archive> - Instead of creating a new one, enable the specified archive

Example 1

[flight@gateway1 ~]$ flight silo migration switch
Enabled archive has been switched to 'debkxnfs'.