Automating NRT Downloads

This guide provides instructions on how to automate NRT data downloads without having to rely on FTP. NRT FTP access has been discontinued

Earthdata URS

ESDIS, our parent organization, requires that we track who downloads files. In addition, MODIS and VIIRS Science teams require that we protect some data from download by the general public but permit it for authorized users.

To meet these requirements, ESDIS has implemented Earthdata Profile (URS), a profile manager that helps us keep a record of anyone using our services.

Previously, NRT would ask users to authenticate via a username & password dialog. Now, NRT redirects users to the URS login screen. URS will redirect users back to NRT upon successful authentication.

For scripted downloads we recommend using MODAPS Download Tokens in conjunction with wget. Instructions can be found below.

Deprecation of FTP

File Transfer Protocol (FTP) has been around for many generations and has been used successfully for transfering files via FTP clients and numerous scripting languages. Unfortunately, FTP is also considered a security risk by many cybersecurity experts.

NASA’s Goddard Space Flight Center has blocked all requests to public facing FTP servers—including LAADS DAAC and LANCE NRT.

Downloading via HTTP

Hypertext Transfer Protocol (HTTP) is the protocol that drives most web site internet traffic today. A variant of the protocol, called “HTTPS”, “S” for “Secure”, has been chosen to replace FTP.

HTTPS encrypts all transactions between client and server. This makes it extremely difficult for third parties to intercept what is being transferred.

All HTTPS downloads will require either an active login session or an Download Token (i.e. token) passed in via an Authorization HTTP header. NRT currently requires HTTPS downloading of all data.

Earthdata Profile

Each user must create an Earthdata URS profile in order to download files. Each URS profile is tracked in NRT by email address, not URS usernames.

Download Tokens

Users wishing to download via a browser simply need to log in and click links within the NRT Archive. Scripted downloads will need to use Earth Data Download (EDL) Tokens in order to be properly authorized.

Requesting a Download Token

Any user that does not already have a Download Token will need to create them following the steps below:

  1. Log in by going to Profile -> Login
  2. Navigate to Profile -> Download Tokens
  3. Create your new Download Token by clicking the "Generate Download Token" button

Retrieving a Download Token

Any forgotten or lost Download Tokens can be replaced by simply logging in and generating a new token.

How to use a Download Token

This example uses the curl command to make a request for the file on our web server at the URL https://nrt4.modaps.eosdis.nasa.gov/api/v2/content/archives/PATH_TO_MY_FILE

curl -v -H "Authorization: Bearer MY_APP_KEY" "-C - https://nrt4.modaps.eosdis.nasa.gov/api/v2/content/archives/PATH_TO_MY_FILE" > result

The example above passes your Download Token via the Authorization HTTP header while utilizing the Bearer schema. When finished, the resulting download will be written to a file called “result“ in whatever directory (folder) you run the command from.

The Download Token located in the header is how NRT identifies users. If the Download Token is invalid, missing, or an -h is used instead of -H, the curl command will not work. The -v parameter tells curl to be verbose so extra information about the request and response will be printed out. If this extra information is not needed the -v parameter can be omitted.

Automation

If all you need is one file and you know which file it is, it is much easier to go to the NRT archive and click to download as needed. If you need many files you might prefer to rely on scripts. We have samples for wget, an open source batch downloader.

Code Samples

wget

wget is an open source utility that can recursively download entire directories of files with just one command. The only path that is required is the root directory. wget will automatically traverse the directory and download any files it locates.

wget is free and available for Linux, macOS, and Windows.

Installation

  1. Linux
    1. Launch a command-line terminal
    2. Type yum install wget -y
  2. macOS
    1. Install Homebrew (admin privileges required)
    2. Launch Applications > Utiliites > Terminal
    3. Type brew install wget
  3. Windows
    1. Download the latest 32-bit or 64-bit binary (.exe) for your system
    2. Move it to C:\Windows\System32 (admin privileges will be required)
    3. Click the Windows Menu > Run
    4. Type cmd and hit Enter
    5. In the Windows Command Prompt, type wget -h to verify the binary is being executed successfully
    6. If any errors appear, the wget.exe binary may be not be located in correct directory or you may need to switch from 32-bit <-> 64-bit

Command-Line/Terminal usage:

wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=4 "https://nrt4.modaps.eosdis.nasa.gov/api/v2/content/archives/PATH_TO_DATA_DIRECTORY" --header "Authorization: Bearer INSERT_YOUR_TOKEN_HERE" -P TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM

macOS/Linux example:

wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=4 "https://nrt4.modaps.eosdis.nasa.gov/api/v2/content/archives/allData/6/MOD09/2018/143/" --header "Authorization: Bearer ABCDEFGH-12345-IJKL-6789-MNOPQRST" -P /Users/jdoe/data

Windows example:

wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=4 "https://nrt4.modaps.eosdis.nasa.gov/api/v2/content/archives/allData/6/MOD09/2018/143/" --header "Authorization: Bearer ABCDEFGH-12345-IJKL-6789-MNOPQRST" -P C:\Users\jdoe\data

Be sure to replace the following:

  • PATH_TO_DATA_DIRECTORY: location of source directory in NRT Archive such as allData/6/MOD09/2018/143/
  • INSERT_YOUR_TOKEN_HERE: Your token
  • TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM: Where you would like to download the files. Examples include /Users/jdoe/data for macOS and Linux or C:\Users\jdoe\data for Windows

Please contact MODAPS Support with any issues regarding NRT file downloads.