I would like to add my metagenome data to a new volume I have created. I did this by following the ‘Creating and attaching volumes using Horizon’. Now I have created one I do not know how to add my files to it. Ideally I would like to upload files using FileZilla, is this possible? If so, how do I connect (what info)?
Once I have added my data how do I call to it in a vm it is attached to? Also how do I share access to this volume so other users of my custom group vm can access my files too?
If you have been using FileZilla before, using your new volume is as simple as transferring data into a directory. Once your volume is mounted, files can be placed into the directory that the volume is mounted at and they will occupy space on the volume, not the root disk of the instance.
If you are using a custom instance (i.e. not GVL), please follow the FileZilla manual for instructions for connecting to a remote server here (follow the “On a Mac” section).
If you would like to give multiple people access to this VM (and data), you can approach it in one of two ways:
Share a user account (ubuntu). All users can and work on the same files, with all of its associated upsides and downsides i.e. its easy, but other users can clobber your files. Make a good backup of your data beforehand, make regular backups of analysis results to your local computer, and be careful who is working on what data. To allow another user access to the ubuntu user account, just append the new user’s public SSH key to the bottom of /home/ubuntu/.ssh/authorized_keys. They can then access the instance in the same way as you: ssh -i ~/.ssh/privkey.pem ubuntu@<ip_address>
Make a user account for each person using the adduser command. There’s significantly more administrative overhead using this approach - you have to set file permissions correctly so that everyone can r+w+x appropriately. Please read the guide for the adduser command above for instructions on how to create a new user. Once you’ve created a new user account, append the new user’s public key to the bottom of /home/<new_user>/.ssh/authorized_keys. They can then access the instance using the new user account: ssh -i ~/.ssh/privkey.pem <new_user>@<ip_address>. Finally, set file permissions on data that you’d like to share - this is a reasonable primer.
Ah ok, so I first need to mount the volume to an instance, then I can transfer files via FileZilla to volume directory within that instance?
In giving multiple access to others, I am guessing we cannot all be on the same one at once? If not, I guess I can make up to 10 identical custom VMs for as many people to access individually. In that case, would I need to make a volume for each and mount them to respective VMs, or can these different custom VMs access the data on one single volume at the same time? (I am guessing not but it would avoid extra work if I can…!)
If I would need to create multiple volumes (one per VM) then could I not just add the data to each instance directly, assuming there is not a huge amount of it?
If it is not already clear from my many hundreds of questions, I would like to demo the analysis I am doing on my metagenomic data for a group of interested peers later this week. Much of this has been on my mac / local HPC resources, but climb obviously offers a more elegant way for folks without the laptop compute / HPC access to become accustomed to this kind of analysis…
A single volume can only be attached to one VM. A single VM can have multiple volumes attached to it.
To save transferring data more than once, you can snapshot your data volume and then create a clone of your original volume from the snapshot via the advanced interface. This new volume can then be attached to a new instance.
You could approach it this way, but snapshots are more convenient, because you only have to transfer the data TO CLIMB once. The rest of the heavy-lifting data transfers are handled by us.
I did kinda guess this - thanks for spreading the word! We can also help with providing multiple identical VMs for tutorial sessions and workshops if you give us some notice - just send me a PM.
Thanks that’s really good to know - I am planning a similar thing for my own group later in the year so will definitely be in touch about that!
Regarding accessing my volume in FileZilla, should I expect to see the volume as a directory by the name I gave it? I see from my advanced interface that the volume I want to add data to is ‘Attached to: /dev/sda on standys’ but I do not have a directory called ‘sda’ at /dev… How do I get to the volume?
Following the volume mounting tutorial - when you get to the mount stage, i.e.
mount /dev/sdX /home/ubuntu/example
This specifies that the DEVICE (/dev/sdX) should be mounted (and therefore accessible) at /home/ubuntu/example.
In this case, putting files into the directory /home/ubuntu/example, would place them on the new volume.
Also, its unlikely that your newly-created volume will have a device label of /dev/sda - this is reserved for the root disk of the VM and is created automatically when the VM is started. Your first new volume will have a device label of /dev/sdb, your second /dev/sdc, and so on.