Deep Learning Memo

Introduction

When traning a neural network, there is lot’s of details that require attention, like how to set the learning rate, how to plot the fig, how to manage your project, etc. These details have no relations with the fundamental basis or the first principle of machine learning, they are just a sort of words of experience.

For green hands, there exists a huge gap between simply comprehending and copying other peoples neural networks from coding one’s own neural network. It’s a tough but effective process. Thus, I want to record my road of practicing deep learning by updating my Blog. This passage won’t discuss the basic principle and only involve my own experiences and insights during my practical process.

If you want to update this blog with your own experiences, just contact with the author.

20250310 Using matplotlib in Linux

When using matplotlib in my WSL subsystem, I encountered errors as follows:

qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, minimal, minimalegl, offscreen, vnc, webgl, xcb.

[1]    5870 IOT instruction (core dumped)  python try_plotting.py

It seems that Linux cannot load the Qt platform plugin, which as a result failed to load the images.

Solutions

We cannot let Linux to directly show the plotted figure in the terminal for its lack of GUI interface. But we can realize it by changing Matplotlib’s backend:

import matplotlib as mpl
mpl.use('Agg')
#...
#...
import matplotlib.plt as plt
# forbid scientific notation
np.set_printoptions(suppress=True)


# Then showing the figure by saving the figures.

plt.savefig("save/img.png")

Of course, you can use juypter notebook instead. Remember! Use plt.close() if you want to save more than one image.

20250310 Arranging my file-structure

The contents below is generated by deepseek-v3.

Simplified Project Structure

your_project/
│
├── data/                    # Raw and processed data
│   ├── raw/                 
│   └── processed/           
│
├── models/                  # Trained models
│   ├── checkpoints/         # Model checkpoints
│   └── final/               # Final models
│
├── outputs/                 # Training outputs
│   ├── logs/                # Logs (e.g., TensorBoard)
│   ├── images/              # Generated images
│   └── plots/               # Plots
│
├── scripts/                 # Scripts
│   ├── train.py             # Training script
│   └── utils.py             # Utility functions
│
└── README.md                # Project description

Code to Auto-Create Directories

import os
from datetime import datetime

# Base directory
base_dir = "your_project"

# Subdirectories
dirs = {
    "data": ["raw", "processed"],
    "models": ["checkpoints", "final"],
    "outputs": ["logs", "images", "plots"],
    "scripts": []
}

# Create directories
for parent_dir, sub_dirs in dirs.items():
    parent_path = os.path.join(base_dir, parent_dir)
    os.makedirs(parent_path, exist_ok=True)
    for sub_dir in sub_dirs:
        sub_path = os.path.join(parent_path, sub_dir)
        os.makedirs(sub_path, exist_ok=True)

# Timestamped output directory
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
output_dir = os.path.join(base_dir, "outputs", timestamp)
os.makedirs(output_dir, exist_ok=True)

print(f"Directories created. Output directory: {output_dir}")

Usage in Training Script

import os
import matplotlib.pyplot as plt

# Example output directory
output_dir = "your_project/outputs/20231025_123456"

# Save model checkpoint
model_checkpoint_path = os.path.join(output_dir, "model_checkpoint.pth")
# torch.save(model.state_dict(), model_checkpoint_path)  # PyTorch example

# Save training log
log_file_path = os.path.join(output_dir, "training_log.txt")
with open(log_file_path, "w") as f:
    f.write("Epoch 1, Loss: 0.123\n")

# Save generated image
image_path = os.path.join(output_dir, "training_plot.png")
plt.plot([1, 2, 3], [4, 5, 6])
plt.savefig(image_path)
plt.close()

print(f"Training outputs saved to: {output_dir}")

Additional Tips

1. Use a Config File:

   # config.py
   BASE_DIR = "your_project"
   DATA_DIR = os.path.join(BASE_DIR, "data")
   MODELS_DIR = os.path.join(BASE_DIR, "models")
   OUTPUTS_DIR = os.path.join(BASE_DIR, "outputs")

2. Use Logging:

   import logging
   logging.basicConfig(filename=os.path.join(output_dir, "training.log"), level=logging.INFO)
   logging.info("Epoch 1, Loss: 0.123")

3. Version Control:
   Add to .gitignore:

   your_project/data/
   your_project/models/
   your_project/outputs/