Welcome to the documentation of ITRA ! 🎈

ITRA (abbreviation for Image Text Representation Alignment) is a codebase for flexible and efficient vision language learning. ITRA features a unified interface to easily access state-of-the-art pretrained models, adapters, loss functions from various sources.

ITRA supports training, evaluation and benchmarking on a rich variety of tasks, including zero-shot/k-NN/linear classification, retrieval, word embedding and sentence embedding evaluation. In the meantime, ITRA is also highly modular extensible and configurable, facilitating future development and customization.

Important

ITRA is an ongoing project developing by the Artificial Intelligence of Multi-modality Group (AIM Group, https://multimodality.group) at Hohai University lead by Prof. Fan Liu. A temporary repository of the codebase is located at: https://github.com/ChenDelong1999/ITRA

Note

If you find any bugs or have any recommendations for building ITRA, please raise a issue in the repo, thanks~

Introduction

• About This Codebase
  • Model Builder
  • Training Objectives
  • Downstream Evaluation
• Related Work
  • OpenCLIP (LAION AI)
  • LAVIS - LAnguage VISion (Salesforce Research)
  • MMF (Facebook AI Research)
  • X-modaler (JD AI Research)
  • TorchMultimodal (Facebook AI Research)
  • Multimodal-Toolkit (Georgian)
  • 贵司公开 Codebase
• Change Log
  • V0.0.1

Getting Started

• Install Dependencies
• Prepare Data
  • Image-text Pairs Dataset from CSV file
  • MS COCO Captions dataset
  • Image Classification Dataset
  • SentEval Datasets
  • EVEVATER Image Classification Datasets
  • NORI Datasets on OSS (for Megvii Useres)

User Guide

• Load Pretrained Multi-modal Weights
  • From OpenCLIP
  • From ChineseCLIP
  • From Taiyi-CLIP
• Load Pretrained Uni-modal Weights
  • Image Backbone
    • From Torchvision
    • From Torch Hub
  • Text Backbone
    • From HuggingFace🤗Transformers
    • From Sentence Transformers
• Custom Training Data
  • Episodic Training
  • Combining Multiple Datasets
• Loss Functions
• Use Adapters
• Freeze Model Parameters During Training

Example Usage

• Evaluate Pretrained Models
  • Zero-shot Image-text Retrieval
  • Zero-shot Image Classification
  • Linear Probing and KNN CClassification
  • Clustering Evaluation
  • Sentence Embedding Evaluation
  • ELEVATOR Image Classification Benchmark
• CLIP Pretraining
  • Standard Contrastive Language Image Pretraining From Scratch
  • Train a Tiny CLIP
• Fine-tuning CLIP for MS-COCO Retrieval
  • Getting Started: Naive Fine-tuning Baseline
  • Tuning Hyper-parameters
  • Scaling up Batch Size by Partially Freeze Weights
  • More Tricks for Fine-tuning
    • Layer-wise Learning Rate Decay (LLDR)
    • Exponential Moving Average (EMA)
    • Wise-FT. Evaluate the model with weight space ensemble
    • rsicd retrieval
• Image Classification (UniCL)
  • Train an Image Classification Model From scratch
  • Fine-tuning CLIP for ImageNet Classification
• Language-to-vision Knowledge Distillation
• Vision-to-language Knowledge Distillation

Todo

New features incoming👇

• Refract main.py

• Write help messages for arguments

• Use YAML

• Project

  • install as package

  • Pypi package publishing

• Evaluation reports

  • zero-shot classification

  • linear/knn classification

  • clustering evaluation

  • SentEval

  • word embedding

  • MS Marco retrieval

  • Chinese CLIPs’ Evaluation Reports (ImageNet-CN zero-shot, MC-COCO-CN retrieval)

• Implementations

  • UniCL-based image classification

  • Validate loss functions

  • Validate Adapters

  • SimCSE and PromptBERT re-implementation

  • Vision-to-language Knowledge Distillation

  • Language-to-vision Knowledge Distillation

  • Teacher selection based on Information Bottleneck Theory