Singularity is installed on our HPCC. However, you may want to develop your own containers first on a local machine.
Many HPC centers including MSU HPCC do not allow Docker containers through Docker. However, Singularity is compatible with Docker, and you can use Docker containers through Singularity. There are a few distinct difference between Docker and Singularity.
Docker:
- Inside a Docker image, the user's privilege is escalated to root on the host system. This privilege is not supported by most HPCC including MSU HPCC. It means that Docker will not be installed on our system.
Singularity:
- User has root privileges if elevated with "sudo" when a container runs.
- Can run and modify Docker images and containers
These key difference make Singularity be installed on most HPCC. In addition, virtually all Docker containers can be run through Singularity, users can effectively run Docker on MSU HPCC.
Installation
Singularity exits as two major version, 2 and 3. Current version on MSU HPCC is 3.5.3. Therefore, in this tutorial, I will use version 3.
To Install Singularity on your local machine, click here:
https://www.sylabs.io/guides/3.0/user-guide/installation.html#installation
- The version of singularity on MSU HPCC is currently 3.4.1. The official documentation for this version is at https://www.sylabs.io/guides/3.4/user-guide/index.html. All singularity commands are built into the system such as 'singularity shell' and 'singularity exec', which means you can invoke these commands directly from the command line.
Check Installation
When you install Singularity on your local machine, then you can check the installation with
$ singularity pull shub://vsoch/hello-world INFO: Downloading shub image 59.75 MiB / 59.75 MiB [========================================================================================] 100.00% 10.46 MiB/s 5s
In the above example, I used the Singularity Hub “unique resource identifier,” or uri, "shub://" which tells the software to run an image from Singularity Hub.
To get help, you can use the help command which gives a general overview of Singularity options and subcommands as follows:
$ singularity --help
Linux container platform optimized for High Performance Computing (HPC) and
Enterprise Performance Computing (EPC)
Usage:
singularity [global options...]
Description:
Singularity containers provide an application virtualization layer enabling
mobility of compute via both application and environment portability. With
Singularity one is capable of building a root file system that runs on any
other Linux system where Singularity is installed.
Options:
-d, --debug print debugging information (highest verbosity)
-h, --help help for singularity
--nocolor print without color output (default False)
-q, --quiet suppress normal output
-s, --silent only print errors
-v, --verbose print additional information
--version version for singularity
Available Commands:
build Build a Singularity image
cache Manage the local cache
capability Manage Linux capabilities for users and groups
config Manage various singularity configuration (root user only)
delete Deletes requested image from the library
exec Run a command within a container
help Help about any command
inspect Show metadata for an image
instance Manage containers running as services
key Manage OpenPGP keys
oci Manage OCI containers
plugin Manage Singularity plugins
pull Pull an image from a URI
push Upload image to the provided URI
remote Manage singularity remote endpoints
run Run the user-defined default command within a container
run-help Show the user-defined help for an image
search Search a Container Library for images
shell Run a shell within a container
sif siftool is a program for Singularity Image Format (SIF) file manipulation
sign Attach a cryptographic signature to an image
test Run the user-defined tests within a container
verify Verify cryptographic signatures attached to an image
version Show the version for Singularity
Examples:
$ singularity help <command> [<subcommand>]
$ singularity help build
$ singularity help instance start
For additional help or support, please visit https://www.sylabs.io/docs/
You can use the help command if you want to see the information about subcommands. For example, to see the pull command help,
$ singularity help pullPull an image from a URI Usage: singularity pull [pull options...] [output file] <URI> Description: The 'pull' command allows you to download or build a container from a given URI. Supported URIs include: library: Pull an image from the currently configured library library://user/collection/container[:tag] docker: Pull an image from Docker Hub docker://user/image:tag shub: Pull an image from Singularity Hub shub://user/image:tag oras: Pull a SIF image from a supporting OCI registry oras://registry/namespace/image:tag http, https: Pull an image using the http(s?) protocol https://library.sylabs.io/v1/imagefile/library/default/alpine:latest Options: --arch string architecture to pull from library (default "amd64") --dir string download images to the specific directory --disable-cache dont use cached images/blobs and dont create them --docker-login login to a Docker Repository interactively -F, --force overwrite an image file if it exists -h, --help help for pull --library string download images from the provided library (default "https://library.sylabs.io") --no-cleanup do NOT clean up bundle after failed build, can be helpul for debugging --nohttps do NOT use HTTPS with the docker:// transport (useful for local docker registries without a certificate) Examples: From Sylabs cloud library $ singularity pull alpine.sif library://alpine:latest From Docker $ singularity pull tensorflow.sif docker://tensorflow/tensorflow:latest From Shub $ singularity pull singularity-images.sif shub://vsoch/singularity-images From supporting OCI registry (e.g. Azure Container Registry) $ singularity pull image.sif oras://<username>.azurecr.io/namespace/image:tag For additional help or support, please visit https://www.sylabs.io/docs/
Downloading pre-built images
I already downloaded a pre-built image "hello-world" from shub, one of the registries, using pull command. This is the easiest way to use Singularity.
You can use the pull command to download pre-built images from a number of Container Registries, here we’ll be focusing on the Singularity-Hub or DockerHub. The following are some of container registries.
- library - images hosted on Sylabs Cloud
- shub - images hosted on Singularity Hub
- docker - images hosted on Docker Hub
- localimage - images saved on your machine
- yum - yum based systems such as CentOS and Scientific Linux
- debootstrap - apt based systems such as Debian and Ubuntu
- arch - Arch Linux
- busybox - BusyBox
- zypper - zypper based systems such as Suse and OpenSuse
Pulling an images from Sylabs cloud library
In this example, I will pull a base Alpine container from Sylabs cloud:
$ singularity pull library://sylabsed/linux/alpine INFO: Downloading library image 2.08 MiB / 2.08 MiB [===========================================================================================] 100.00% 4.74 MiB/s 0s
You can rename the container using the –name flag:
$ singularity pull --name my_alpine.sif library://sylabsed/linux/alpine INFO: Downloading library image 2.08 MiB / 2.08 MiB [===========================================================================================] 100.00% 9.65 MiB/s 0s
The above example will save the image as "my_alpine.sif"
Pulling an images from Docker hub
This example pulls an Alpine image from Docker hub
$ singularity pull docker://alpine INFO: Converting OCI blobs to SIF format INFO: Starting build... Getting image source signatures Copying blob df20fa9351a1 done Copying config 0f5f445df8 done Writing manifest to image destination Storing signatures 2020/08/20 15:53:52 info unpack layer: sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c INFO: Creating SIF file... INFO: Build complete: alpine_latest.sif
Interact with images
You can interact with images with shell, exec, and run commands.
To learn how to interact with images, let's first pull an image "lolcow_latest.sif" from the libray.
$ singularity pull library://sylabsed/examples/lolcow
shell
The shell command allows you to spawn a new shell within your container and interact with it as if it is a virtual machine.
$singularity shell lolcow_latest.sif Singularity>
The change in prompt indicates that you have entered the container. Once inside of a container, you are the same user as you are on the host system.
Singularity>whoami choiyj
To exit from a container, type exit.
Singularity>exit $
exec
The exec command allows you to execute a custom command within a container by specifying the image file. For instance, to execute the cowsay program within the lolcow_latest.sif container:
$ singularity exec lolcow_latest.sif cowsay container camp rocks
______________________
< container camp rocks >
----------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
You can also use shell command to run the program in the container.
Singularity> cowsay container camp rocks
______________________
< container camp rocks >
----------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
run
Singularity containers contain runscripts. These are user defined scripts which define the actions of a container when user runs it. The runscript can be performed with the run command, or simply by calling the container as though it were an executable.
$ singularity run lolcow_latest.sif
_________________________________________
/ You're ugly and your mother dresses you \
\ funny. /
-----------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Cache setting
By default, Singularity uses a temporary directory to save Docker files as tarballs:
$ ls ~/.singularity cache/ docker/ metadata/ $ ls .singularity/docker/ sha256:0efe07335a049e6afcd757db2d17ba37a12b717eb807acb03ddf3cd756b9fc2a.tar.gz sha256:43a98c1873995475a895f3d79f405232ef5230076b3f610c949c2e8341743af7.tar.gz sha256:6b1bb01b3a3b72463ae8ac5666d57b28f1a21d5256271910ac8df841aa04ecd1.tar.gz sha256:c2ca09a1934b951505ecc4d6b2e4ab7f9bf27bcdfb8999d0181deca74daf7683.tar.gz sha256:d6c3619d2153ffdefa4a9c19f15c5d566ce271b397a84537baa9ee45b24178f2.tar.gz
You can change these by theses cache directories by specifying the location on your localhost as following:
$ mkdir -p $SCRATCH/singularity_tmp $ mkdir -p $SCRATCH/singularity_scratch $ SINGULARITY_TMPDIR=$SCRATCH/singularity_scratch SINGULARITY_CACHEDIR=$SCRATCH/singularity_tmp singularity --debug pull --name ubuntu-tmpdir.sif docker://ubuntu
Creating writable containers with --sandbox options
If you want to build a container within a writable directory (called a sandbox), you can do that with --sandbox option. We’ve already covered how you can pull an existing container from Docker Hub, but we can also build a Singularity container from docker using the build command:
$ singularity build --sandbox ubuntu-latest/ docker://ubuntu
With --sandbox option, you can changes and write files within the directory, but those changes will not remain when you finished using the container. To make those changes persistent, use --writable option when you start a container.
$ singularity shell --writable ubuntu-latest/