Page tree
Skip to end of metadata
Go to start of metadata

Singularity is installed on our HPCC. However, you may want to develop your own containers first on a local machine.

Many HPC centers including MSU HPCC do not allow Docker containers through Docker. However, Singularity is compatible with Docker, and you can use Docker containers through Singularity. There are a few distinct difference between Docker and Singularity.

Docker:

  • Inside a Docker image, the user's privilege is escalated to root on the host system. This privilege is not supported by most HPCC including MSU HPCC. It means that Docker will not be installed on our system. 

Singularity:

  • User has root privileges if elevated with "sudo" when a container runs.
  • Can run and modify Docker images and containers

 These key difference make Singularity be installed on most HPCC. In addition, virtually all Docker containers can be run through Singularity, users can effectively run Docker on MSU HPCC.

Installation

Singularity exits as two major version, 2 and 3. Current version on MSU HPCC is 3.5.3. Therefore, in this tutorial, I will use version 3.

To Install Singularity on your local machine, click here: 

https://www.sylabs.io/guides/3.0/user-guide/installation.html#installation

  • The version of singularity on MSU HPCC is currently 3.4.1. The official documentation for this version is at https://www.sylabs.io/guides/3.4/user-guide/index.htmlAll singularity commands are built into the system such as 'singularity shell' and 'singularity exec', which means you can invoke these commands directly from the command line.

Check Installation

When you install Singularity on your local machine, then you can check the installation with

$ singularity pull shub://vsoch/hello-world
INFO:    Downloading shub image
 59.75 MiB / 59.75 MiB [========================================================================================] 100.00% 10.46 MiB/s 5s

In the above example, I used the Singularity Hub “unique resource identifier,” or uri, "shub://" which tells the software to run an image from Singularity Hub.

To get help, you can use the help command which gives a general overview of Singularity options and subcommands as follows:

$ singularity --help

Linux container platform optimized for High Performance Computing (HPC) and
Enterprise Performance Computing (EPC)

Usage:
  singularity [global options...]

Description:
  Singularity containers provide an application virtualization layer enabling
  mobility of compute via both application and environment portability. With
  Singularity one is capable of building a root file system that runs on any
  other Linux system where Singularity is installed.

Options:
  -d, --debug     print debugging information (highest verbosity)
  -h, --help      help for singularity
      --nocolor   print without color output (default False)
  -q, --quiet     suppress normal output
  -s, --silent    only print errors
  -v, --verbose   print additional information
      --version   version for singularity

Available Commands:
  build       Build a Singularity image
  cache       Manage the local cache
  capability  Manage Linux capabilities for users and groups
  config      Manage various singularity configuration (root user only)
  delete      Deletes requested image from the library
  exec        Run a command within a container
  help        Help about any command
  inspect     Show metadata for an image
  instance    Manage containers running as services
  key         Manage OpenPGP keys
  oci         Manage OCI containers
  plugin      Manage Singularity plugins
  pull        Pull an image from a URI
  push        Upload image to the provided URI
  remote      Manage singularity remote endpoints
  run         Run the user-defined default command within a container
  run-help    Show the user-defined help for an image
  search      Search a Container Library for images
  shell       Run a shell within a container
  sif         siftool is a program for Singularity Image Format (SIF) file manipulation
  sign        Attach a cryptographic signature to an image
  test        Run the user-defined tests within a container
  verify      Verify cryptographic signatures attached to an image
  version     Show the version for Singularity

Examples:
  $ singularity help <command> [<subcommand>]
  $ singularity help build
  $ singularity help instance start


For additional help or support, please visit https://www.sylabs.io/docs/

You can use the help command if you want to see the information about subcommands. For example, to see the pull command help,

$ singularity help pullPull an image from a URI

Usage:
singularity pull [pull options...] [output file] <URI>

Description:
The 'pull' command allows you to download or build a container from a given
URI. Supported URIs include:

library: Pull an image from the currently configured library
library://user/collection/container[:tag]

docker: Pull an image from Docker Hub
docker://user/image:tag

shub: Pull an image from Singularity Hub
shub://user/image:tag

oras: Pull a SIF image from a supporting OCI registry
oras://registry/namespace/image:tag

http, https: Pull an image using the http(s?) protocol
https://library.sylabs.io/v1/imagefile/library/default/alpine:latest

Options:
--arch string architecture to pull from library (default "amd64")
--dir string download images to the specific directory
--disable-cache dont use cached images/blobs and dont create them
--docker-login login to a Docker Repository interactively
-F, --force overwrite an image file if it exists
-h, --help help for pull
--library string download images from the provided library
(default "https://library.sylabs.io")
--no-cleanup do NOT clean up bundle after failed build, can be
helpul for debugging
--nohttps do NOT use HTTPS with the docker:// transport
(useful for local docker registries without a
certificate)


Examples:
From Sylabs cloud library
$ singularity pull alpine.sif library://alpine:latest

From Docker
$ singularity pull tensorflow.sif docker://tensorflow/tensorflow:latest

From Shub
$ singularity pull singularity-images.sif shub://vsoch/singularity-images

From supporting OCI registry (e.g. Azure Container Registry)
$ singularity pull image.sif oras://<username>.azurecr.io/namespace/image:tag


For additional help or support, please visit https://www.sylabs.io/docs/

Downloading pre-built images

I already downloaded a pre-built image "hello-world" from shub, one of the registries, using pull command. This is the easiest way to use Singularity.

You can use the pull command to download pre-built images from a number of Container Registries, here we’ll be focusing on the Singularity-Hub or DockerHub. The following are some of container registries.

  • library - images hosted on Sylabs Cloud
  • shub - images hosted on Singularity Hub
  • docker - images hosted on Docker Hub
  • localimage - images saved on your machine
  • yum - yum based systems such as CentOS and Scientific Linux
  • debootstrap - apt based systems such as Debian and Ubuntu
  • arch - Arch Linux
  • busybox - BusyBox
  • zypper - zypper based systems such as Suse and OpenSuse

Pulling an images from Sylabs cloud library

In this example, I will pull a base Alpine container from Sylabs cloud:

$ singularity pull library://sylabsed/linux/alpine
INFO:    Downloading library image
 2.08 MiB / 2.08 MiB [===========================================================================================] 100.00% 4.74 MiB/s 0s

You can rename the container using the –name flag:

$ singularity pull --name my_alpine.sif library://sylabsed/linux/alpine
INFO:    Downloading library image
 2.08 MiB / 2.08 MiB [===========================================================================================] 100.00% 9.65 MiB/s 0s

The above example will save the image as "my_alpine.sif"

Pulling an images from Docker hub

This example pulls an Alpine image from Docker hub

$ singularity pull docker://alpine
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob df20fa9351a1 done
Copying config 0f5f445df8 done
Writing manifest to image destination
Storing signatures
2020/08/20 15:53:52  info unpack layer: sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c
INFO:    Creating SIF file...
INFO:    Build complete: alpine_latest.sif

Interact with images

You can interact with images with shell, exec, and run commands.

To learn how to interact with images, let's first pull an image "lolcow_latest.sif" from the libray.

$ singularity pull library://sylabsed/examples/lolcow

shell

The shell command allows you to spawn a new shell within your container and interact with it as if it is a virtual machine.

$singularity shell lolcow_latest.sif
  Singularity>

The change in prompt indicates that you have entered the container. Once inside of a container, you are the same user as you are on the host system.

Singularity>whoami
choiyj

To exit from a container, type exit.

Singularity>exit
$

exec

The exec command allows you to execute a custom command within a container by specifying the image file. For instance, to execute the cowsay program within the lolcow_latest.sif container:

$ singularity exec lolcow_latest.sif cowsay container camp rocks
______________________
< container camp rocks >
 ----------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

You can also use shell command to run the program in the container.

Singularity>  cowsay container camp rocks
 ______________________
< container camp rocks >
 ----------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

run

Singularity containers contain runscripts. These are user defined scripts which define the actions of a container when user runs it. The runscript can be performed with the run command, or simply by calling the container as though it were an executable.

$ singularity run lolcow_latest.sif
 _________________________________________
/ You're ugly and your mother dresses you \
\ funny.                                  /
 -----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Cache setting

By default, Singularity uses a temporary directory to save Docker files as tarballs:

$ ls ~/.singularity
cache/  docker/  metadata/
$ ls .singularity/docker/
sha256:0efe07335a049e6afcd757db2d17ba37a12b717eb807acb03ddf3cd756b9fc2a.tar.gz
sha256:43a98c1873995475a895f3d79f405232ef5230076b3f610c949c2e8341743af7.tar.gz
sha256:6b1bb01b3a3b72463ae8ac5666d57b28f1a21d5256271910ac8df841aa04ecd1.tar.gz
sha256:c2ca09a1934b951505ecc4d6b2e4ab7f9bf27bcdfb8999d0181deca74daf7683.tar.gz
sha256:d6c3619d2153ffdefa4a9c19f15c5d566ce271b397a84537baa9ee45b24178f2.tar.gz

You can change these by theses cache directories by specifying the location on your localhost as following:

$ mkdir -p $SCRATCH/singularity_tmp
$ mkdir -p $SCRATCH/singularity_scratch
$ SINGULARITY_TMPDIR=$SCRATCH/singularity_scratch SINGULARITY_CACHEDIR=$SCRATCH/singularity_tmp singularity --debug pull --name ubuntu-tmpdir.sif docker://ubuntu

Creating writable containers with --sandbox options

If you want to build a container within a writable directory (called a sandbox), you can do that with --sandbox option. We’ve already covered how you can pull an existing container from Docker Hub, but we can also build a Singularity container from docker using the build command:

$ singularity build --sandbox ubuntu-latest/  docker://ubuntu

With --sandbox option, you can changes and write files within the directory, but those changes will not remain when you finished using the container. To make those changes persistent, use --writable option when you start a container.

$ singularity shell --writable ubuntu-latest/
  • No labels