AWS - S3 ストレージクラス

S3 ストレージクラス

ストレージクラス S3 Standard(S3 標準) S3 Intelligent-Tiering S3 Standard-IA(S3 標準-IA) S3 One Zone-IA (S3 1ゾーン-IA) S3 Glacier Instant Retrieval S3 Glacier Flexible Retrieval S3 Glacier Deep Archive
AZ 3つ以上のAZ 3つ以上のAZ 3つ以上のAZ 1つのAZ 3つ以上のAZ 3つ以上のAZ
想定されるデータタイプ 頻繁にアクセスされるアクティブデータ アクセスパターンが変化するデータ アクセス頻度が低いデータ 再⽣可能でアクセス頻度が低いデータ めったにアクセスされないデータ アーカイブデータ ⻑期保存のアーカイブデータ
設計上の耐久性 99.999999999% 99.999999999% 99.999999999% 99.999999999% 99.999999999% 99.999999999% 99.999999999%
設計上の可用性 99.99% 99.9% 99.9% 99.5% 99.9% 99.99% 99.99%
可用性(SLA) 99.9% 99% 99% 99% 99% 99.9% 99.9%
レイテンシー ミリ秒単位のアクセス ミリ秒単位のアクセス ミリ秒単位のアクセス ミリ秒単位のアクセス ミリ秒単位のアクセス 分から時間単位の復元(数分〜12時間) 時間単位の復元(12〜48時間)
取り出し料金 - - GBあたり GBあたり GBあたり GBあたり GBあたり
最低保存期間 - - 30日 30日 90日 90日 180日
最小オブジェクトサイズ - - 128KB 128KB 128KB 40KB 40KB
ストレージ価格 0.025 〜0.023 USD/GB ⽉ 0.025 〜0.002 USD/GB ⽉ 0.0138 USD/GB ⽉ 0.011 USD/GB ⽉ 0.005 USD/GB ⽉ 0.0045 USD/GB ⽉ 0.002 USD/GB ⽉

※ストレージ価格は2021/12 東京リージョン価格

docker-compose DB系

Oracle

https://github.com/oracle/docker-images

1
2
3
git clone https://github.com/oracle/docker-images.git
cd docker-images/OracleDatabase/SingleInstance/dockerfiles
./buildDockerImage.sh -v 18.4.0 -x
1
2
3
4
5
6
7
8
9
10
11
12
13
version: '3.8'
services:

db:
image: oracle/database:18.4.0-xe
ports:
- 1521:1521
- 5500:5500
environment:
- ORACLE_PWD=xxxxxxxxxxxx
volumes:
- ./db-data:/opt/oracle/oradata
- ./db-script:/opt/oracle/scripts/setup

MariaDB

https://hub.docker.com/_/mariadb

1
2
3
4
5
6
7
8
9
10
11
12
13
version: '3'
services:

mariadb:
container_name: mariadb
image: mariadb:10.5
ports:
- "3306:3306"
environment:
- MYSQL_ROOT_PASSWORD=xxxxxxxxxxxx
volumes:
- ./db-data:/var/lib/mysql
- ./db-script:/docker-entrypoint-initdb.d

Microsoft SQL Server

https://hub.docker.com/_/microsoft-mssql-server

1
2
3
4
5
6
7
8
9
10
11
12
13
14
version: '3'
services:
mssql:
image: mcr.microsoft.com/mssql/server:2019-latest
container_name: mssql
ports:
- "1433:1433"
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=xxxxxxxxxxxx
- MSSQL_PID=Developer
- MSSQL_LCID=1041
- MSSQL_COLLATION=Japanese_CI_AS


PostgreSQL

https://hub.docker.com/_/postgres

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
version: '3'

services:

postgres:
build:
context: .
dockerfile: ./dockerfiles/postgresql/Dockerfile
args:
- DB_LANG=ja_JP
image: postgres:13
container_name: "postgres"
ports:
- "5432:5432"
environment:
POSTGRES_PASSWORD: xxxxxxxxxxxx
POSTGRES_INITDB_ARGS: "--encoding=UTF-8 --locale=C"
volumes:
- ./db-data:/var/lib/postgresql/data
- ./db-script:/docker-entrypoint-initdb.d

pgadmin4:
image: dpage/pgadmin4:latest
container_name: pgadmin4
ports:
- "8000:80"
environment:
PGADMIN_DEFAULT_EMAIL: [email protected]
PGADMIN_DEFAULT_PASSWORD: xxxxxxxxxxxx
hostname: pgadmin4
depends_on:
- postgres

Redis

https://hub.docker.com/_/redis

1
2
3
4
5
6
7
8
version: '3'
services:
redis:
image: "redis:latest"
ports:
- "6379:6379"
volumes:
- "./data/redis:/data"

PyTorch Torchvision Models

Model [email protected] [email protected]
AlexNet 56.522 79.066
VGG-11 69.020 88.628
VGG-13 69.928 89.246
VGG-16 71.592 90.382
VGG-19 72.376 90.876
VGG-11 with batch normalization 70.370 89.810
VGG-13 with batch normalization 71.586 90.374
VGG-16 with batch normalization 73.360 91.516
VGG-19 with batch normalization 74.218 91.842
ResNet-18 69.758 89.078
ResNet-34 73.314 91.420
ResNet-50 76.130 92.862
ResNet-101 77.374 93.546
ResNet-152 78.312 94.046
SqueezeNet 1.0 58.092 80.420
SqueezeNet 1.1 58.178 80.624
Densenet-121 74.434 91.972
Densenet-169 75.600 92.806
Densenet-201 76.896 93.370
Densenet-161 77.138 93.560
Inception v3 77.294 93.450
GoogleNet 69.778 89.530
ShuffleNet V2 x1.0 69.362 88.316
ShuffleNet V2 x0.5 60.552 81.746
MobileNet V2 71.878 90.286
MobileNet V3 Large 74.042 91.340
MobileNet V3 Small 67.668 87.402
ResNeXt-50-32x4d 77.618 93.698
ResNeXt-101-32x8d 79.312 94.526
Wide ResNet-50-2 78.468 94.086
Wide ResNet-101-2 78.848 94.284
MNASNet 1.0 73.456 91.510
MNASNet 0.5 67.734 87.490
EfficientNet-B0 77.692 93.532
EfficientNet-B1 78.642 94.186
EfficientNet-B2 80.608 95.310
EfficientNet-B3 82.008 96.054
EfficientNet-B4 83.384 96.594
EfficientNet-B5 83.444 96.628
EfficientNet-B6 84.008 96.916
EfficientNet-B7 84.122 96.908
regnet_x_400mf 72.834 90.950
regnet_x_800mf 75.212 92.348
regnet_x_1_6gf 77.040 93.440
regnet_x_3_2gf 78.364 93.992
regnet_x_8gf 79.344 94.686
regnet_x_16gf 80.058 94.944
regnet_x_32gf 80.622 95.248
regnet_y_400mf 74.046 91.716
regnet_y_800mf 76.420 93.136
regnet_y_1_6gf 77.950 93.966
regnet_y_3_2gf 78.948 94.576
regnet_y_8gf 80.032 95.048
regnet_y_16gf 80.424 95.240
regnet_y_32gf 80.878 95.340

Alexnet

https://arxiv.org/abs/1404.5997

  • torchvision.models.alexnet

VGG

https://arxiv.org/abs/1409.1556

  • torchvision.models.vgg11

  • torchvision.models.vgg11_bn

  • torchvision.models.vgg13

  • torchvision.models.vgg13_bn

  • torchvision.models.vgg16

  • torchvision.models.vgg16_bn

  • torchvision.models.vgg19

  • torchvision.models.vgg19_bn


ResNet

https://arxiv.org/abs/1512.03385

  • torchvision.models.resnet18
  • torchvision.models.resnet34
  • torchvision.models.resnet50
  • torchvision.models.resnet101
  • torchvision.models.resnet152

SqueezeNet

https://arxiv.org/abs/1602.07360

  • torchvision.models.squeezenet1_0
  • torchvision.models.squeezenet1_1

DenseNet

https://arxiv.org/abs/1608.06993

  • torchvision.models.densenet121
  • torchvision.models.densenet169
  • torchvision.models.densenet161
  • torchvision.models.densenet201

Inception v3

https://arxiv.org/abs/1512.00567

  • torchvision.models.inception_v3
    requires scipy to be installed

GoogLeNet

https://arxiv.org/abs/1409.4842

  • torchvision.models.googlenet
    requires scipy to be installed

ShuffleNet v2

https://arxiv.org/abs/1807.11164

  • torchvision.models.shufflenet_v2_x0_5
  • torchvision.models.shufflenet_v2_x1_0
  • torchvision.models.shufflenet_v2_x1_5
  • torchvision.models.shufflenet_v2_x2_0

MobileNet v2

https://arxiv.org/abs/1801.04381

  • torchvision.models.mobilenet_v2

MobileNet v3

https://arxiv.org/abs/1905.02244

  • torchvision.models.mobilenet_v3_large
  • torchvision.models.mobilenet_v3_small

ResNext

https://arxiv.org/abs/1611.05431

  • torchvision.models.resnext50_32x4d
  • torchvision.models.resnext101_32x8d

Wide ResNet

  • torchvision.models.wide_resnet50_2
  • torchvision.models.wide_resnet101_2

MNASNet

https://arxiv.org/abs/1807.11626

  • torchvision.models.mnasnet0_5
  • torchvision.models.mnasnet0_75
  • torchvision.models.mnasnet1_0
  • torchvision.models.mnasnet1_3

EfficientNet

https://arxiv.org/abs/1905.11946

  • torchvision.models.efficientnet_b0
  • torchvision.models.efficientnet_b1
  • torchvision.models.efficientnet_b2
  • torchvision.models.efficientnet_b3
  • torchvision.models.efficientnet_b4
  • torchvision.models.efficientnet_b5
  • torchvision.models.efficientnet_b6
  • torchvision.models.efficientnet_b7

RegNet

https://arxiv.org/abs/2003.13678

  • torchvision.models.regnet_y_400mf
  • torchvision.models.regnet_y_800mf
  • torchvision.models.regnet_y_1_6gf
  • torchvision.models.regnet_y_3_2gf
  • torchvision.models.regnet_y_8gf
  • torchvision.models.regnet_y_16gf
  • torchvision.models.regnet_y_32gf
  • torchvision.models.regnet_x_400mf
  • torchvision.models.regnet_x_800mf
  • torchvision.models.regnet_x_1_6gf
  • torchvision.models.regnet_x_3_2gf
  • torchvision.models.regnet_x_8gf
  • torchvision.models.regnet_x_16gf

WSL Ubuntu 20.04 に Caffe をインストール

http://caffe.berkeleyvision.org/

システムパッケージ更新

1
2
3
4
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get autoremove --purge -y
reboot

必須パッケージインストール

1
2
3
4
5
6
sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev
sudo apt-get install libopencv-dev libboost-all-dev libhdf5-serial-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
sudo apt-get install protobuf-compiler
sudo apt-get install libatlas-base-dev libopenblas-dev

OpenCV3インストール

https://docs.opencv.org/3.4.16/d7/d9f/tutorial_linux_install.html

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#必須パッケージ
sudo apt-get install cmake libgtk2.0-dev pkg-config
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev
sudo apt-get install python-dev python-numpy libtbb2 libtbb-dev
sudo apt-get install libjpeg-dev libpng-dev libtiff-dev libdc1394-22-dev

#ダウンロード
cd /tmp
wget https://github.com/opencv/opencv/archive/3.4.16.zip

#解凍
unzip 3.4.16.zip
cd opencv-3.4.16

#ビルドインストール
mkdir build && cd build
cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local ..
make -j8 && sudo make install

Caffeインストール

1
2
3
4
5
6
7
8
9
10
11
12
cd ~

#ダウンロード
wget https://github.com/BVLC/caffe/archive/refs/tags/1.0.zip

#解凍
unzip 1.0.zip
cd caffe-1.0

#Makefile修正
mv Makefile.config.example Makefile.config
nano Makefile.config

OpenCVバージョン変更

1
2
# Ver 2 -> 3
OPENCV_VERSION := 3

CUDA 11.4以降の場合3.5未満削除

1
2
3
4
5
6
CUDA_ARCH := -gencode arch=compute_35,code=sm_35 \
-gencode arch=compute_50,code=sm_50 \
-gencode arch=compute_52,code=sm_52 \
-gencode arch=compute_60,code=sm_60 \
-gencode arch=compute_61,code=sm_61 \
-gencode arch=compute_61,code=compute_61

ATLASはマルチスレッドCPUを十分にサポートしていないため、行列演算ライブラリをOpenBLASに変更

1
BLAS := open

hdf5のヘッダー・ライブラリパス追加

1
2
3
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib \
/usr/lib/x86_64-linux-gnu/hdf5/serial

ビルド、テスト、インストール

1
make all -j8 && make -j8 && make runtest -j8