tetradat

Description

Package tetradat (TEnsor TRain ADversarial ATtacks) for generation of adversarial examples for artificial neural networks from computer vision domain using tensor train (TT) decomposition and PROTES optimizer based on it. Please, see teneva and PROTES repositories for details.

Installation

1. Install anaconda package manager with python;

2. Create a virtual environment:

   conda create --name tetradat python=3.10 -y

3. Activate the environment:

   conda activate tetradat

4. Download this repo (optional):

   git clone https://github.com/AndreiChertkov/tetradat.git && cd tetradat

5. Install LLaVA framework (optional):

   git clone https://github.com/haotian-liu/LLaVA.git && cd LLaVA && pip install -e . && cd ./../

   In the case of problems, remove in llava/model/__init__.py the block try before installing.

6. Install other dependencies for LLaVA (optional):

   pip install aiohttp pyarrow tensorboardx && pip install transformers==4.37.2

   You may also need pip install fsspec[http]==2024.2.0

7. Install other dependencies:

   pip install numpy matplotlib requests urllib3 protes==0.3.7 torchattacks==3.4.0 sentence-transformers diffusers

8. Delete virtual environment at the end of the work (optional):

   conda activate && conda remove --name tetradat --all -y

Usage

Attacks on computer vision models

Run python manager.py ARGS, then see the outputs in the terminal and results in the result folder. Before starting the new calculation, you can completely delete or rename the result folder; a new result folder will be created automatically in this case.

The calls with the following ARGS may be performed:

• To check the data:

  • python manager.py --task check --kind data --data imagenet

• To check the models:

  • python manager.py --task check --kind model --data imagenet --model alexnet
  • python manager.py --task check --kind model --data imagenet --model googlenet
  • python manager.py --task check --kind model --data imagenet --model inception
  • python manager.py --task check --kind model --data imagenet --model mobilenet
  • python manager.py --task check --kind model --data imagenet --model resnet
  • python manager.py --task check --kind model --data imagenet --model vgg

• To run attacks with the proposed TETRADAT method:

  • python manager.py --task attack --kind attr --data imagenet --model googlenet --model_attr vgg

    You may use any of the models from above here. We select the helper model for attribution by model_attr argument.

• To run attacks with the baselines:

  • python manager.py --task attack --kind bs_onepixel --data imagenet --model googlenet --model_attr vgg

    You may use any of the models from above here; and bs_square or bs_pixle. We set model_attr argument above to skip the images for which the vgg model fails.

• To run the demo attack with the proposed TETRADAT method:

  • python manager.py --task check --kind demo --data imagenet

• To present the computation results:

  • python show.py

    Data for tables will be in the console output and images will be in result/_show folder.

Attacks on multimodal LLava model

This is a rough implementation of an attack on a multimodal model, the code may be unstable.

• Run clear && python sim_wrapper.py for demo of text similarity calculation
• Run clear && python llava_wrapper.py for demo of LLaVa usage
• Run clear && python manager.py --task attack_llava --kind attr --data imagenet --model vgg --model_attr vgg --opt_d 2500 --opt_m 1000 --opt_sc 0.5 --opt_k 50 --opt_k_top 5 --opt_k_gd 10 --opt_lr 1.E-2

Authors

• Andrei Chertkov
• Ivan Oseledets