Download Data to Jarvislabs.ai instances
You can download data to your Jarvislabs.ai GPU/CPU powered instances in multiple ways. Lets looks at some options.
- Wget (Simple and fast)
- Using Curl Wget
- Downloading from Kaggle
- SFTP Transfer from your local directory
Downloading using wget
Open Terminal from JupyterLab, and use wget
+ link
to download any publicly available datasets.
# Downloading CIFAR10 dataset
wget https://s3.amazonaws.com/fast-ai-sample/cifar10.tgz
Downloading using CurlWget
Steps to download data from Kaggle Dataset, Google Drive or from other data sources to Jarvislabs.ai instance using CurlWget extension.
Downloading data from Kaggle
Kaggle provides an API which helps you to
- Download Kaggle datasets
- Upload to Kaggle datasets - Useful, when you want to upload your model
- Make Kaggle submissions
Interested in learning more about Kaggle API check the docs here.
Install Kaggle API
Open terminal.
pip install kaggle --upgrade
Setup Kaggle API
In Kaggle page go to the Account
tab of your user profile and select Create API Token
. This will trigger the download of kaggle.json
, a file containing your API credentials. Place this file in the location ~/.kaggle/kaggle.json
.
If you have uploaded using JupyterLab, use the below commands to copy it to the required location.
mkdir .kaggle
mv kaggle.json .kaggle/
chmod 600 ~/.kaggle/kaggle.json
Downloading Kaggle dataset
Open terminal, and replace feedback-prize-2021
with the competition you are participating.
kaggle competitions download -c feedback-prize-2021
Uploading Kaggle dataset
You can use Kaggle API to upload datasets from Jarvislabs.ai instance to Kaggle datasets.
kaggle datasets init -p /path/to/dataset
You will find dataset-metadata.json file inside the dataset folder. Change the 'id' and the 'title' in the file dataset-metadata.json
kaggle datasets create -p /path/to/dataset
Update an existing dataset
kaggle datasets version -p /path/to/dataset -m "Updated data"
Download Data from Instance to Local using SFTP
SSH
You have to connect your instance with your local terminal.
To do that you have to generate a SSH key in your JarvisLabs instance. Check out this document for more information.
After that press the SSH button on your instance to copy the SSH key and port number of that instance.
The key will be in this format.
ssh -o StrictHostKeyChecking=no -p <port_number> root@sshh.jarvislabs.ai
Paste the key in your terminal in the following format.
sftp -P <port_number> root@sshk.jarvislabs.ai
Download
In your terminal you can use this command to safely transfer anything from your instance to your local directory.
get /path/to/file
Upload
To upload a file from your local to remote instance
put /path/to/upload/file /path/to/upload/destination