Training on GCP

Google has made the deep learning training and deployment process streamlined by allowing us to train models, store them, and deploy them with minimal code. Before training in the cloud, we should first train the model locally to confirm everything works as intended.

To begin, we must set some environment variables and organize our project files into the correct structure required by Cloud ML. Inside the training folder in this chapter’s GitHub repository, you will find the correct file layout. The __init__.py file simply tells GCP that our folder contains an executable Python package.

Next, define the job directory and training data path:

JOB_DIR="model_folder"
TRAIN_DATA="/path/to/data"
  

Now train the model locally using Cloud ML’s local runner:

gcloud ml-engine local train \
    --job-dir $JOB_DIR \
    --module-name trainer.task \
    --package-path trainer/ \
    -- --train-file $TRAIN_DATA
  

If your model trains successfully locally, you are ready for cloud training.

Cloud Training Setup

Training on Cloud ML is submitted as a job. The platform automatically sets up the environment, runs the training, and cleans up afterward.

Each training job must have a unique name. We will call our job classifier_model.

Instead of a local job directory, we now point Cloud ML to a folder inside our Google Cloud Storage bucket.

Choose the closest region to your location in the United States:

• us-west1
• us-central1
• us-east1

JOB_NAME="classifier_model"
JOB_DIR="gs://classifier_bucket123/$JOB_NAME"
PACKAGE_STAGING_PATH="gs://classifier_bucket123"
MAIN_TRAINER_MODULE="trainer.simple_classifier"
REGION="us-east1"
TRAIN_DATA="gs://classifier_bucket123/creditcard.csv"
  

Submit the Cloud Training Job

Send the job to Cloud ML with:

gcloud ml-engine jobs submit training $JOB_NAME \
    --staging-bucket $PACKAGE_STAGING_PATH \
    --job-dir $JOB_DIR \
    --package-path /trainer \
    --module-name $MAIN_TRAINER_MODULE \
    --runtime-version 1.8 \
    --region $REGION \
    -- --train-file $TRAIN_DATA
  

The --runtime-version must match your installed version of TensorFlow. You can check your version by running:

python -c "import tensorflow as tf; print(tf.__version__)"
  

Monitor Training Progress

After submitting the job, Cloud ML will confirm that training has started. You can log in to the GCP console and use the Jobs section to check:

• Training metrics
• Error messages
• Output logs
• Resource usage

This information provides the same visibility you'd normally see when training locally.

When training is complete, the script will output the SavedModel binaries to your Google Cloud Storage bucket. If you have reached this stage successfully, you are ready to deploy the model as an API.