學(xué)習(xí)前言
一起來(lái)看看Efficientdet的keras實(shí)現(xiàn)吧，順便訓(xùn)練一下自己的數(shù)據(jù)。
什么是Efficientdet目標(biāo)檢測(cè)算法
最近，谷歌大腦 Mingxing Tan、Ruoming Pang 和 Quoc V. Le 提出新架構(gòu) EfficientDet，結(jié)合 EfficientNet（同樣來(lái)自該團(tuán)隊(duì)）和新提出的 BiFPN，實(shí)現(xiàn)新的 SOTA 結(jié)果。
源碼下載
https://github.com/bubbliiiing/efficientdet-keras
喜歡的可以點(diǎn)個(gè)star噢。
Efficientdet實(shí)現(xiàn)思路
一、預(yù)測(cè)部分
1、主干網(wǎng)絡(luò)介紹
Efficientdet采用Efficientnet作為主干特征提取網(wǎng)絡(luò)。EfficientNet-B0對(duì)應(yīng)Efficientdet-D0；EfficientNet-B1對(duì)應(yīng)Efficientdet-D1；以此類(lèi)推。
EfficientNet模型具有很獨(dú)特的特點(diǎn)，這個(gè)特點(diǎn)是參考其它優(yōu)秀神經(jīng)網(wǎng)絡(luò)設(shè)計(jì)出來(lái)的。經(jīng)典的神經(jīng)網(wǎng)絡(luò)特點(diǎn)如下：
1、利用殘差神經(jīng)網(wǎng)絡(luò)增大神經(jīng)網(wǎng)絡(luò)的深度，通過(guò)更深的神經(jīng)網(wǎng)絡(luò)實(shí)現(xiàn)特征提取。
2、改變每一層提取的特征層數(shù)，實(shí)現(xiàn)更多層的特征提取，得到更多的特征，提升寬度。
3、通過(guò)增大輸入圖片的分辨率也可以使得網(wǎng)絡(luò)可以學(xué)習(xí)與表達(dá)的東西更加豐富，有利于提高精確度。
EfficientNet就是將這三個(gè)特點(diǎn)結(jié)合起來(lái)，通過(guò)一起縮放baseline模型（MobileNet中就通過(guò)縮放α實(shí)現(xiàn)縮放模型，不同的α有不同的模型精度，α=1時(shí)為baseline模型；ResNet其實(shí)也是有一個(gè)baseline模型，在baseline的基礎(chǔ)上通過(guò)改變圖片的深度實(shí)現(xiàn)不同的模型實(shí)現(xiàn)），同時(shí)調(diào)整深度、寬度、輸入圖片的分辨率完成一個(gè)優(yōu)秀的網(wǎng)絡(luò)設(shè)計(jì)。
在EfficientNet模型中，其使用一組固定的縮放系數(shù)統(tǒng)一縮放網(wǎng)絡(luò)深度、寬度和分辨率。
假設(shè)想使用 2N倍的計(jì)算資源，我們可以簡(jiǎn)單的對(duì)網(wǎng)絡(luò)深度擴(kuò)大αN倍、寬度擴(kuò)大βN 、圖像尺寸擴(kuò)大γN倍，這里的α,β,γ都是由原來(lái)的小模型上做微小的網(wǎng)格搜索決定的常量系數(shù)。
如圖為EfficientNet的設(shè)計(jì)思路，從三個(gè)方面同時(shí)拓充網(wǎng)絡(luò)的特性。
本博客以Efficientnet-B0和Efficientdet-D0為例，進(jìn)行Efficientdet的解析。
Efficientnet-B0由1個(gè)Stem+16個(gè)大Blocks堆疊構(gòu)成，16個(gè)大Blocks可以分為1、2、2、3、3、4、1個(gè)Block。Block的通用結(jié)構(gòu)如下，其總體的設(shè)計(jì)思路是Inverted residuals結(jié)構(gòu)和殘差結(jié)構(gòu)，在3x3或者5x5網(wǎng)絡(luò)結(jié)構(gòu)前利用1x1卷積升維，在3x3或者5x5網(wǎng)絡(luò)結(jié)構(gòu)后增加了一個(gè)關(guān)于通道的注意力機(jī)制，最后利用1x1卷積降維后增加一個(gè)大殘差邊。
整體結(jié)構(gòu)如下：
最終獲得三個(gè)有效特征層傳入到BIFPN當(dāng)中進(jìn)行下一步的操作。
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import os
import json
import math
import string
import collections
import numpy as np
from keras import backend
from six.moves import xrange
from nets.layers import BatchNormalization
from keras import layers
BASE_WEIGHTS_PATH = (
'https://github.com/Callidior/keras-applications/'
'releases/download/efficientnet/')
WEIGHTS_HASHES = {
'efficientnet-b0': ('163292582f1c6eaca8e7dc7b51b01c61'
'5b0dbc0039699b4dcd0b975cc21533dc',
'c1421ad80a9fc67c2cc4000f666aa507'
'89ce39eedb4e06d531b0c593890ccff3'),
'efficientnet-b1': ('d0a71ddf51ef7a0ca425bab32b7fa7f1'
'6043ee598ecee73fc674d9560c8f09b0',
'75de265d03ac52fa74f2f510455ba64f'
'9c7c5fd96dc923cd4bfefa3d680c4b68'),
'efficientnet-b2': ('bb5451507a6418a574534aa76a91b106'
'f6b605f3b5dde0b21055694319853086',
'433b60584fafba1ea3de07443b74cfd3'
'2ce004a012020b07ef69e22ba8669333'),
'efficientnet-b3': ('03f1fba367f070bd2545f081cfa7f3e7'
'6f5e1aa3b6f4db700f00552901e75ab9',
'c5d42eb6cfae8567b418ad3845cfd63a'
'a48b87f1bd5df8658a49375a9f3135c7'),
'efficientnet-b4': ('98852de93f74d9833c8640474b2c698d'
'b45ec60690c75b3bacb1845e907bf94f',
'7942c1407ff1feb34113995864970cd4'
'd9d91ea64877e8d9c38b6c1e0767c411'),
'efficientnet-b5': ('30172f1d45f9b8a41352d4219bf930ee'
'3339025fd26ab314a817ba8918fefc7d',
'9d197bc2bfe29165c10a2af8c2ebc675'
'07f5d70456f09e584c71b822941b1952'),
'efficientnet-b6': ('f5270466747753485a082092ac9939ca'
'a546eb3f09edca6d6fff842cad938720',
'1d0923bb038f2f8060faaf0a0449db4b'
'96549a881747b7c7678724ac79f427ed'),
'efficientnet-b7': ('876a41319980638fa597acbbf956a82d'
'10819531ff2dcb1a52277f10c7aefa1a',
'60b56ff3a8daccc8d96edfd40b204c11'
'3e51748da657afd58034d54d3cec2bac')
}
BlockArgs = collections.namedtuple('BlockArgs', [
'kernel_size', 'num_repeat', 'input_filters', 'output_filters',
'expand_ratio', 'id_skip', 'strides', 'se_ratio'
])
# defaults will be a public argument for namedtuple in Python 3.7
# https://docs.python.org/3/library/collections.html#collections.namedtuple
BlockArgs.__new__.__defaults__ = (None,) * len(BlockArgs._fields)
DEFAULT_BLOCKS_ARGS = [
BlockArgs(kernel_size=3, num_repeat=1, input_filters=32, output_filters=16,
expand_ratio=1, id_skip=True, strides=[1, 1], se_ratio=0.25),
BlockArgs(kernel_size=3, num_repeat=2, input_filters=16, output_filters=24,
expand_ratio=6, id_skip=True, strides=[2, 2], se_ratio=0.25),
BlockArgs(kernel_size=5, num_repeat=2, input_filters=24, output_filters=40,
expand_ratio=6, id_skip=True, strides=[2, 2], se_ratio=0.25),
BlockArgs(kernel_size=3, num_repeat=3, input_filters=40, output_filters=80,
expand_ratio=6, id_skip=True, strides=[2, 2], se_ratio=0.25),
BlockArgs(kernel_size=5, num_repeat=3, input_filters=80, output_filters=112,
expand_ratio=6, id_skip=True, strides=[1, 1], se_ratio=0.25),
BlockArgs(kernel_size=5, num_repeat=4, input_filters=112, output_filters=192,
expand_ratio=6, id_skip=True, strides=[2, 2], se_ratio=0.25),
BlockArgs(kernel_size=3, num_repeat=1, input_filters=192, output_filters=320,
expand_ratio=6, id_skip=True, strides=[1, 1], se_ratio=0.25)
]
CONV_KERNEL_INITIALIZER = {
'class_name': 'VarianceScaling',
'config': {
'scale': 2.0,
'mode': 'fan_out',
# EfficientNet actually uses an untruncated normal distribution for
# initializing conv layers, but keras.initializers.VarianceScaling use
# a truncated distribution.
# We decided against a custom initializer for better serializability.
'distribution': 'normal'
}
}
DENSE_KERNEL_INITIALIZER = {
'class_name': 'VarianceScaling',
'config': {
'scale': 1. / 3.,
'mode': 'fan_out',
'distribution': 'uniform'
}
}
def get_swish():
def swish(x):
return x * backend.sigmoid(x)
return swish
def get_dropout():
class FixedDropout(layers.Dropout):
def _get_noise_shape(self, inputs):
if self.noise_shape is None:
return self.noise_shape
symbolic_shape = backend.shape(inputs)
noise_shape = [symbolic_shape[axis] if shape is None else shape
for axis, shape in enumerate(self.noise_shape)]
return tuple(noise_shape)
return FixedDropout
def round_filters(filters, width_coefficient, depth_divisor):
filters *= width_coefficient
new_filters = int(filters + depth_divisor / 2) // depth_divisor * depth_divisor
new_filters = max(depth_divisor, new_filters)
if new_filters 0.9 * filters:
new_filters += depth_divisor
return int(new_filters)
def round_repeats(repeats, depth_coefficient):
return int(math.ceil(depth_coefficient * repeats))
def mb_conv_block(inputs, block_args, activation, drop_rate=None, prefix='', freeze_bn=False):
has_se = (block_args.se_ratio is not None) and (0 block_args.se_ratio = 1)
bn_axis = 3
Dropout = get_dropout()
filters = block_args.input_filters * block_args.expand_ratio
if block_args.expand_ratio != 1:
x = layers.Conv2D(filters, 1,
padding='same',
use_bias=False,
kernel_initializer=CONV_KERNEL_INITIALIZER,
name=prefix + 'expand_conv')(inputs)
x = layers.BatchNormalization(axis=bn_axis, name=prefix + 'expand_bn')(x)
x = layers.Activation(activation, name=prefix + 'expand_activation')(x)
else:
x = inputs
# Depthwise Convolution
x = layers.DepthwiseConv2D(block_args.kernel_size,
strides=block_args.strides,
padding='same',
use_bias=False,
depthwise_initializer=CONV_KERNEL_INITIALIZER,
name=prefix + 'dwconv')(x)
x = layers.BatchNormalization(axis=bn_axis, name=prefix + 'bn')(x)
x = layers.Activation(activation, name=prefix + 'activation')(x)
# Squeeze and Excitation phase
if has_se:
num_reduced_filters = max(1, int(
block_args.input_filters * block_args.se_ratio
))
se_tensor = layers.GlobalAveragePooling2D(name=prefix + 'se_squeeze')(x)
target_shape = (1, 1, filters) if backend.image_data_format() == 'channels_last' else (filters, 1, 1)
se_tensor = layers.Reshape(target_shape, name=prefix + 'se_reshape')(se_tensor)
se_tensor = layers.Conv2D(num_reduced_filters, 1,
activation=activation,
padding='same',
use_bias=True,
kernel_initializer=CONV_KERNEL_INITIALIZER,
name=prefix + 'se_reduce')(se_tensor)
se_tensor = layers.Conv2D(filters, 1,
activation='sigmoid',
padding='same',
use_bias=True,
kernel_initializer=CONV_KERNEL_INITIALIZER,
name=prefix + 'se_expand')(se_tensor)
if backend.backend() == 'theano':
# For the Theano backend, we have to explicitly make
# the excitation weights broadcastable.
pattern = ([True, True, True, False] if backend.image_data_format() == 'channels_last'
else [True, False, True, True])
se_tensor = layers.Lambda(
lambda x: backend.pattern_broadcast(x, pattern),
name=prefix + 'se_broadcast')(se_tensor)
x = layers.multiply([x, se_tensor], name=prefix + 'se_excite')
# Output phase
x = layers.Conv2D(block_args.output_filters, 1,
padding='same',
use_bias=False,
kernel_initializer=CONV_KERNEL_INITIALIZER,
name=prefix + 'project_conv')(x)
# x = BatchNormalization(freeze=freeze_bn, axis=bn_axis, name=prefix + 'project_bn')(x)
x = layers.BatchNormalization(axis=bn_axis, name=prefix + 'project_bn')(x)
if block_args.id_skip and all(
s == 1 for s in block_args.strides
) and block_args.input_filters == block_args.output_filters:
if drop_rate and (drop_rate > 0):
x = Dropout(drop_rate,
noise_shape=(None, 1, 1, 1),
name=prefix + 'drop')(x)
x = layers.add([x, inputs], name=prefix + 'add')
return x
def EfficientNet(width_coefficient,
depth_coefficient,
default_resolution,
dropout_rate=0.2,
drop_connect_rate=0.2,
depth_divisor=8,
blocks_args=DEFAULT_BLOCKS_ARGS,
model_name='efficientnet',
include_top=True,
weights='imagenet',
input_tensor=None,
input_shape=None,
pooling=None,
classes=1000,
freeze_bn=False,
**kwargs):
features = []
if input_tensor is None:
img_input = layers.Input(shape=input_shape)
else:
img_input = input_tensor
bn_axis = 3
activation = get_swish(**kwargs)
# Build stem
x = img_input
x = layers.Conv2D(round_filters(32, width_coefficient, depth_divisor), 3,
strides=(2, 2),
padding='same',
use_bias=False,
kernel_initializer=CONV_KERNEL_INITIALIZER,
name='stem_conv')(x)
# x = BatchNormalization(freeze=freeze_bn, axis=bn_axis, name='stem_bn')(x)
x = layers.BatchNormalization(axis=bn_axis, name='stem_bn')(x)
x = layers.Activation(activation, name='stem_activation')(x)
# Build blocks
num_blocks_total = sum(block_args.num_repeat for block_args in blocks_args)
block_num = 0
for idx, block_args in enumerate(blocks_args):
assert block_args.num_repeat > 0
# Update block input and output filters based on depth multiplier.
block_args = block_args._replace(
input_filters=round_filters(block_args.input_filters,
width_coefficient, depth_divisor),
output_filters=round_filters(block_args.output_filters,
width_coefficient, depth_divisor),
num_repeat=round_repeats(block_args.num_repeat, depth_coefficient))
# The first block needs to take care of stride and filter size increase.
drop_rate = drop_connect_rate * float(block_num) / num_blocks_total
x = mb_conv_block(x, block_args,
activation=activation,
drop_rate=drop_rate,
prefix='block{}a_'.format(idx + 1),
freeze_bn=freeze_bn
)
block_num += 1
if block_args.num_repeat > 1:
# pylint: disable=protected-access
block_args = block_args._replace(
input_filters=block_args.output_filters, strides=[1, 1])
# pylint: enable=protected-access
for bidx in xrange(block_args.num_repeat - 1):
drop_rate = drop_connect_rate * float(block_num) / num_blocks_total
block_prefix = 'block{}{}_'.format(
idx + 1,
string.ascii_lowercase[bidx + 1]
)
x = mb_conv_block(x, block_args,
activation=activation,
drop_rate=drop_rate,
prefix=block_prefix,
freeze_bn=freeze_bn
)
block_num += 1
if idx len(blocks_args) - 1 and blocks_args[idx + 1].strides[0] == 2:
features.append(x)
elif idx == len(blocks_args) - 1:
features.append(x)
return features
def EfficientNetB0(include_top=True,
weights='imagenet',
input_tensor=None,
input_shape=None,
pooling=None,
classes=1000,
**kwargs):
return EfficientNet(1.0, 1.0, 224, 0.2,
model_name='efficientnet-b0',
include_top=include_top, weights=weights,
input_tensor=input_tensor, input_shape=input_shape,
pooling=pooling, classes=classes,
**kwargs)
def EfficientNetB1(include_top=True,
weights='imagenet',
input_tensor=None,
input_shape=None,
pooling=None,
classes=1000,
**kwargs):
return EfficientNet(1.0, 1.1, 240, 0.2,
model_name='efficientnet-b1',
include_top=include_top, weights=weights,
input_tensor=input_tensor, input_shape=input_shape,
pooling=pooling, classes=classes,
**kwargs)
def EfficientNetB2(include_top=True,
weights='imagenet',
input_tensor=None,
input_shape=None,
pooling=None,
classes=1000,
**kwargs):
return EfficientNet(1.1, 1.2, 260, 0.3,
model_name='efficientnet-b2',
include_top=include_top, weights=weights,
input_tensor=input_tensor, input_shape=input_shape,
pooling=pooling, classes=classes,
**kwargs)
def EfficientNetB3(include_top=True,
weights='imagenet',
input_tensor=None,
input_shape=None,
pooling=None,
classes=1000,
**kwargs):
return EfficientNet(1.2, 1.4, 300, 0.3,
model_name='efficientnet-b3',
include_top=include_top, weights=weights,
input_tensor=input_tensor, input_shape=input_shape,
pooling=pooling, classes=classes,
**kwargs)
def EfficientNetB4(include_top=True,
weights='imagenet',
input_tensor=None,
input_shape=None,
pooling=None,
classes=1000,
**kwargs):
return EfficientNet(1.4, 1.8, 380, 0.4,
model_name='efficientnet-b4',
include_top=include_top, weights=weights,
input_tensor=input_tensor, input_shape=input_shape,
pooling=pooling, classes=classes,
**kwargs)
def EfficientNetB5(include_top=True,
weights='imagenet',
input_tensor=None,
input_shape=None,
pooling=None,
classes=1000,
**kwargs):
return EfficientNet(1.6, 2.2, 456, 0.4,
model_name='efficientnet-b5',
include_top=include_top, weights=weights,
input_tensor=input_tensor, input_shape=input_shape,
pooling=pooling, classes=classes,
**kwargs)
def EfficientNetB6(include_top=True,
weights='imagenet',
input_tensor=None,
input_shape=None,
pooling=None,
classes=1000,
**kwargs):
return EfficientNet(1.8, 2.6, 528, 0.5,
model_name='efficientnet-b6',
include_top=include_top, weights=weights,
input_tensor=input_tensor, input_shape=input_shape,
pooling=pooling, classes=classes,
**kwargs)
def EfficientNetB7(include_top=True,
weights='imagenet',
input_tensor=None,
input_shape=None,
pooling=None,
classes=1000,
**kwargs):
return EfficientNet(2.0, 3.1, 600, 0.5,
model_name='efficientnet-b7',
include_top=include_top, weights=weights,
input_tensor=input_tensor, input_shape=input_shape,
pooling=pooling, classes=classes,
**kwargs)
2、BiFPN加強(qiáng)特征提取
BiFPN簡(jiǎn)單來(lái)講是一個(gè)加強(qiáng)版本的FPN，上圖是BiFPN，下圖是普通的FPN，大家可以看到，與普通的FPN相比，BiFPN的FPN構(gòu)建更加復(fù)雜，中間還增加了許多連接。
構(gòu)建BiFPN可以分為多步：
1、獲得P3_in、P4_in、P5_in、P6_in、P7_in，通過(guò)主干特征提取網(wǎng)絡(luò)，我們已經(jīng)可以獲得P3、P4、P5，還需要進(jìn)行兩次下采樣獲得P6、P7。
P3、P4、P5在經(jīng)過(guò)1x1卷積調(diào)整通道數(shù)后，就可以作為P3_in、P4_in、P5_in了，在構(gòu)建BiFPN的第一步，需要構(gòu)建兩個(gè)P4_in、P5_in（原版是這樣設(shè)計(jì)的）。
實(shí)現(xiàn)代碼如下：
_, _, C3, C4, C5 = features
# 第一次BIFPN需要 下采樣 與 降通道 獲得 p3_in p4_in p5_in p6_in p7_in
#-----------------------------下采樣 與 降通道----------------------------#
P3_in = C3
P3_in = layers.Conv2D(num_channels, kernel_size=1, padding='same',
name=f'fpn_cells/cell_{id}/fnode3/resample_0_0_8/conv2d')(P3_in)
P3_in = layers.BatchNormalization(momentum=MOMENTUM, epsilon=EPSILON,
name=f'fpn_cells/cell_{id}/fnode3/resample_0_0_8/bn')(P3_in)
P4_in = C4
P4_in_1 = layers.Conv2D(num_channels, kernel_size=1, padding='same',
name=f'fpn_cells/cell_{id}/fnode2/resample_0_1_7/conv2d')(P4_in)
P4_in_1 = layers.BatchNormalization(momentum=MOMENTUM, epsilon=EPSILON,
name=f'fpn_cells/cell_{id}/fnode2/resample_0_1_7/bn')(P4_in_1)
P4_in_2 = layers.Conv2D(num_channels, kernel_size=1, padding='same',
name=f'fpn_cells/cell_{id}/fnode4/resample_0_1_9/conv2d')(P4_in)
P4_in_2 = layers.BatchNormalization(momentum=MOMENTUM, epsilon=EPSILON,
name=f'fpn_cells/cell_{id}/fnode4/resample_0_1_9/bn')(P4_in_2)
P5_in = C5
P5_in_1 = layers.Conv2D(num_channels, kernel_size=1, padding='same',
name=f'fpn_cells/cell_{id}/fnode1/resample_0_2_6/conv2d')(P5_in)
P5_in_1 = layers.BatchNormalization(momentum=MOMENTUM, epsilon=EPSILON,
name=f'fpn_cells/cell_{id}/fnode1/resample_0_2_6/bn')(P5_in_1)
P5_in_2 = layers.Conv2D(num_channels, kernel_size=1, padding='same',
name=f'fpn_cells/cell_{id}/fnode5/resample_0_2_10/conv2d')(P5_in)
P5_in_2 = layers.BatchNormalization(momentum=MOMENTUM, epsilon=EPSILON,
name=f'fpn_cells/cell_{id}/fnode5/resample_0_2_10/bn')(P5_in_2)
P6_in = layers.Conv2D(num_channels, kernel_size=1, padding='same', name='resample_p6/conv2d')(C5)
P6_in = layers.BatchNormalization(momentum=MOMENTUM, epsilon=EPSILON, name='resample_p6/bn')(P6_in)
P6_in = layers.MaxPooling2D(pool_size=3, strides=2, padding='same', name='resample_p6/maxpool')(P6_in)
P7_in = layers.MaxPooling2D(pool_size=3, strides=2, padding='same', name='resample_p7/maxpool')(P6_in)
#-------------------------------------------------------------------------#
2、在獲得P3_in、P4_in_1、P4_in_2、P5_in_1、P5_in_2、P6_in、P7_in之后需要對(duì)P7_in進(jìn)行上采樣，上采樣后與P6_in堆疊獲得P6_td；之后對(duì)P6_td進(jìn)行上采樣，上采樣后與P5_in_1進(jìn)行堆疊獲得P5_td；之后對(duì)P5_td進(jìn)行上采樣，上采樣后與P4_in_1進(jìn)行堆疊獲得P4_td；之后對(duì)P4_td進(jìn)行上采樣，上采樣后與P3_in進(jìn)行堆疊獲得P3_out。
實(shí)現(xiàn)代碼如下：
#--------------------------構(gòu)建BIFPN的上下采樣循環(huán)-------------------------#
P7_U = layers.UpSampling2D()(P7_in)
P6_td = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode0/add')([P6_in, P7_U])
P6_td = layers.Activation(lambda x: tf.nn.swish(x))(P6_td)
P6_td = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode0/op_after_combine5')(P6_td)
P6_U = layers.UpSampling2D()(P6_td)
P5_td = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode1/add')([P5_in_1, P6_U])
P5_td = layers.Activation(lambda x: tf.nn.swish(x))(P5_td)
P5_td = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode1/op_after_combine6')(P5_td)
P5_U = layers.UpSampling2D()(P5_td)
P4_td = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode2/add')([P4_in_1, P5_U])
P4_td = layers.Activation(lambda x: tf.nn.swish(x))(P4_td)
P4_td = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode2/op_after_combine7')(P4_td)
P4_U = layers.UpSampling2D()(P4_td)
P3_out = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode3/add')([P3_in, P4_U])
P3_out = layers.Activation(lambda x: tf.nn.swish(x))(P3_out)
P3_out = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode3/op_after_combine8')(P3_out)
#-------------------------------------------------------------------------#
3、在獲得P3_out、P4_td、P4_in_2、P5_td、P5_in_2、P6_in、P6_td、P7_in之后，之后需要對(duì)P3_out進(jìn)行下采樣，下采樣后與P4_td、P4_in_2堆疊獲得P4_out；之后對(duì)P4_out進(jìn)行下采樣，下采樣后與P5_td、P5_in_2進(jìn)行堆疊獲得P5_out；之后對(duì)P5_out進(jìn)行下采樣，下采樣后與P6_in、P6_td進(jìn)行堆疊獲得P6_out；之后對(duì)P6_out進(jìn)行下采樣，下采樣后與P7_in進(jìn)行堆疊獲得P7_out。
實(shí)現(xiàn)代碼如下：
#--------------------------構(gòu)建BIFPN的上下采樣循環(huán)-------------------------#
P3_D = layers.MaxPooling2D(pool_size=3, strides=2, padding='same')(P3_out)
P4_out = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode4/add')([P4_in_2, P4_td, P3_D])
P4_out = layers.Activation(lambda x: tf.nn.swish(x))(P4_out)
P4_out = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode4/op_after_combine9')(P4_out)
P4_D = layers.MaxPooling2D(pool_size=3, strides=2, padding='same')(P4_out)
P5_out = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode5/add')([P5_in_2, P5_td, P4_D])
P5_out = layers.Activation(lambda x: tf.nn.swish(x))(P5_out)
P5_out = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode5/op_after_combine10')(P5_out)
P5_D = layers.MaxPooling2D(pool_size=3, strides=2, padding='same')(P5_out)
P6_out = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode6/add')([P6_in, P6_td, P5_D])
P6_out = layers.Activation(lambda x: tf.nn.swish(x))(P6_out)
P6_out = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode6/op_after_combine11')(P6_out)
P6_D = layers.MaxPooling2D(pool_size=3, strides=2, padding='same')(P6_out)
P7_out = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode7/add')([P7_in, P6_D])
P7_out = layers.Activation(lambda x: tf.nn.swish(x))(P7_out)
P7_out = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode7/op_after_combine12')(P7_out)
#-------------------------------------------------------------------------#
4、將獲得的P3_out、P4_out、P5_out、P6_out、P7_out作為P3_in、P4_in、P5_in、P6_in、P7_in，重復(fù)2、3步驟進(jìn)行堆疊即可，對(duì)于Effiicientdet B0來(lái)講，還需要重復(fù)2次，需要注意P4_in_1和P4_in_2此時(shí)不需要分開(kāi)了，P5也是。
實(shí)現(xiàn)代碼如下：
P3_in, P4_in, P5_in, P6_in, P7_in = features
P7_U = layers.UpSampling2D()(P7_in)
P6_td = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode0/add')([P6_in, P7_U])
P6_td = layers.Activation(lambda x: tf.nn.swish(x))(P6_td)
P6_td = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode0/op_after_combine5')(P6_td)
P6_U = layers.UpSampling2D()(P6_td)
P5_td = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode1/add')([P5_in, P6_U])
P5_td = layers.Activation(lambda x: tf.nn.swish(x))(P5_td)
P5_td = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode1/op_after_combine6')(P5_td)
P5_U = layers.UpSampling2D()(P5_td)
P4_td = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode2/add')([P4_in, P5_U])
P4_td = layers.Activation(lambda x: tf.nn.swish(x))(P4_td)
P4_td = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode2/op_after_combine7')(P4_td)
P4_U = layers.UpSampling2D()(P4_td)
P3_out = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode3/add')([P3_in, P4_U])
P3_out = layers.Activation(lambda x: tf.nn.swish(x))(P3_out)
P3_out = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode3/op_after_combine8')(P3_out)
P3_D = layers.MaxPooling2D(pool_size=3, strides=2, padding='same')(P3_out)
P4_out = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode4/add')([P4_in, P4_td, P3_D])
P4_out = layers.Activation(lambda x: tf.nn.swish(x))(P4_out)
P4_out = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode4/op_after_combine9')(P4_out)
P4_D = layers.MaxPooling2D(pool_size=3, strides=2, padding='same')(P4_out)
P5_out = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode5/add')([P5_in, P5_td, P4_D])
P5_out = layers.Activation(lambda x: tf.nn.swish(x))(P5_out)
P5_out = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode5/op_after_combine10')(P5_out)
P5_D = layers.MaxPooling2D(pool_size=3, strides=2, padding='same')(P5_out)
P6_out = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode6/add')([P6_in, P6_td, P5_D])
P6_out = layers.Activation(lambda x: tf.nn.swish(x))(P6_out)
P6_out = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode6/op_after_combine11')(P6_out)
P6_D = layers.MaxPooling2D(pool_size=3, strides=2, padding='same')(P6_out)
P7_out = wBiFPNAdd(name=f'fpn_cells/cell_{id}/fnode7/add')([P7_in, P6_D])
P7_out = layers.Activation(lambda x: tf.nn.swish(x))(P7_out)
P7_out = SeparableConvBlock(num_channels=num_channels, kernel_size=3, strides=1,
name=f'fpn_cells/cell_{id}/fnode7/op_after_combine12')(P7_out)
3、從特征獲取預(yù)測(cè)結(jié)果
通過(guò)第二部的重復(fù)運(yùn)算，我們獲得了P3_out, P4_out, P5_out, P6_out, P7_out。
為了和普通特征層區(qū)分，我們稱(chēng)之為有效特征層，將這五個(gè)有效的特征層傳輸過(guò)ClassNet+BoxNet就可以獲得預(yù)測(cè)結(jié)果了。
對(duì)于Efficientdet-B0來(lái)講：
ClassNet采用3次64通道的卷積和1次num_priors x num_classes的卷積，num_priors指的是該特征層所擁有的先驗(yàn)框數(shù)量，num_classes指的是網(wǎng)絡(luò)一共對(duì)多少類(lèi)的目標(biāo)進(jìn)行檢測(cè)。
BoxNet采用3次64通道的卷積和1次num_priors x 4的卷積，num_priors指的是該特征層所擁有的先驗(yàn)框數(shù)量，4指的是先驗(yàn)框的調(diào)整情況。
需要注意的是，每個(gè)特征層所用的ClassNet是同一個(gè)ClassNet；每個(gè)特征層所用的BoxNet是同一個(gè)BoxNet。
其中：
num_priors x 4的卷積 用于預(yù)測(cè) 該特征層上 每一個(gè)網(wǎng)格點(diǎn)上 每一個(gè)先驗(yàn)框的變化情況。**
num_priors x num_classes的卷積 用于預(yù)測(cè) 該特征層上 每一個(gè)網(wǎng)格點(diǎn)上 每一個(gè)預(yù)測(cè)框對(duì)應(yīng)的種類(lèi)。
實(shí)現(xiàn)代碼為：
class BoxNet:
def __init__(self, width, depth, num_anchors=9, freeze_bn=False, name='box_net', **kwargs):
self.name = name
self.width = width
self.depth = depth
self.num_anchors = num_anchors
options = {
'kernel_size': 3,
'strides': 1,
'padding': 'same',
'bias_initializer': 'zeros',
'depthwise_initializer': initializers.VarianceScaling(),
'pointwise_initializer': initializers.VarianceScaling(),
}
self.convs = [layers.SeparableConv2D(filters=width, name=f'{self.name}/box-{i}', **options) for i in range(depth)]
self.head = layers.SeparableConv2D(filters=num_anchors * 4, name=f'{self.name}/box-predict', **options)
self.bns = [
[layers.BatchNormalization(momentum=MOMENTUM, epsilon=EPSILON, name=f'{self.name}/box-{i}-bn-{j}') for j in
range(3, 8)]
for i in range(depth)]
self.relu = layers.Lambda(lambda x: tf.nn.swish(x))
self.reshape = layers.Reshape((-1, 4))
def call(self, inputs):
feature, level = inputs
for i in range(self.depth):
feature = self.convs[i](feature)
feature = self.bns[i][level](feature)
feature = self.relu(feature)
outputs = self.head(feature)
outputs = self.reshape(outputs)
return outputs
class ClassNet:
def __init__(self, width, depth, num_classes=20, num_anchors=9, freeze_bn=False, name='class_net', **kwargs):
self.name = name
self.width = width
self.depth = depth
self.num_classes = num_classes
self.num_anchors = num_anchors
options = {
'kernel_size': 3,
'strides': 1,
'padding': 'same',
'depthwise_initializer': initializers.VarianceScaling(),
'pointwise_initializer': initializers.VarianceScaling(),
}
self.convs = [layers.SeparableConv2D(filters=width, bias_initializer='zeros', name=f'{self.name}/class-{i}',
**options)
for i in range(depth)]
self.head = layers.SeparableConv2D(filters=num_classes * num_anchors,
bias_initializer=PriorProbability(probability=0.01),
name=f'{self.name}/class-predict', **options)
self.bns = [
[layers.BatchNormalization(momentum=MOMENTUM, epsilon=EPSILON, name=f'{self.name}/class-{i}-bn-{j}') for j
in range(3, 8)]
for i in range(depth)]
self.relu = layers.Lambda(lambda x: tf.nn.swish(x))
self.reshape = layers.Reshape((-1, num_classes))
self.activation = layers.Activation('sigmoid')
def call(self, inputs):
feature, level = inputs
for i in range(self.depth):
feature = self.convs[i](feature)
feature = self.bns[i][level](feature)
feature = self.relu(feature)
outputs = self.head(feature)
outputs = self.reshape(outputs)
outputs = self.activation(outputs)
return outputs
4、預(yù)測(cè)結(jié)果的解碼
我們通過(guò)對(duì)每一個(gè)特征層的處理，可以獲得三個(gè)內(nèi)容，分別是：
num_priors x 4的卷積 用于預(yù)測(cè) 該特征層上 每一個(gè)網(wǎng)格點(diǎn)上 每一個(gè)先驗(yàn)框的變化情況。**
num_priors x num_classes的卷積 用于預(yù)測(cè) 該特征層上 每一個(gè)網(wǎng)格點(diǎn)上 每一個(gè)預(yù)測(cè)框對(duì)應(yīng)的種類(lèi)。
每一個(gè)有效特征層對(duì)應(yīng)的先驗(yàn)框?qū)?yīng)著該特征層上 每一個(gè)網(wǎng)格點(diǎn)上 預(yù)先設(shè)定好的9個(gè)框。
我們利用 num_priors x 4的卷積 與 每一個(gè)有效特征層對(duì)應(yīng)的先驗(yàn)框 獲得框的真實(shí)位置。
每一個(gè)有效特征層對(duì)應(yīng)的先驗(yàn)框就是，如圖所示的作用：
每一個(gè)有效特征層將整個(gè)圖片分成與其長(zhǎng)寬對(duì)應(yīng)的網(wǎng)格，如P3的特征層就是將整個(gè)圖像分成64x64個(gè)網(wǎng)格；然后從每個(gè)網(wǎng)格中心建立9個(gè)先驗(yàn)框，一共64x64x9個(gè)，36864個(gè)先驗(yàn)框。
先驗(yàn)框雖然可以代表一定的框的位置信息與框的大小信息，但是其是有限的，無(wú)法表示任意情況，因此還需要調(diào)整，Efficientdet利用3次64通道的卷積+num_priors x 4的卷積的結(jié)果對(duì)先驗(yàn)框進(jìn)行調(diào)整。
num_priors x 4中的num_priors表示了這個(gè)網(wǎng)格點(diǎn)所包含的先驗(yàn)框數(shù)量，其中的4表示了框的左上角xy軸，右下角xy的調(diào)整情況。
Efficientdet解碼過(guò)程就是將對(duì)應(yīng)的先驗(yàn)框的左上角和右下角進(jìn)行位置的調(diào)整，調(diào)整完的結(jié)果就是預(yù)測(cè)框的位置了。
當(dāng)然得到最終的預(yù)測(cè)結(jié)構(gòu)后還要進(jìn)行得分排序與非極大抑制篩選這一部分基本上是所有目標(biāo)檢測(cè)通用的部分。
1、取出每一類(lèi)得分大于confidence_threshold的框和得分。
2、利用框的位置和得分進(jìn)行非極大抑制。
實(shí)現(xiàn)代碼如下：
def decode_boxes(self, mbox_loc, mbox_priorbox):
# 獲得先驗(yàn)框的寬與高
prior_width = mbox_priorbox[:, 2] - mbox_priorbox[:, 0]
prior_height = mbox_priorbox[:, 3] - mbox_priorbox[:, 1]
# 獲得先驗(yàn)框的中心點(diǎn)
prior_center_x = 0.5 * (mbox_priorbox[:, 2] + mbox_priorbox[:, 0])
prior_center_y = 0.5 * (mbox_priorbox[:, 3] + mbox_priorbox[:, 1])
# 真實(shí)框距離先驗(yàn)框中心的xy軸偏移情況
decode_bbox_center_x = mbox_loc[:, 0] * prior_width * 0.1
decode_bbox_center_x += prior_center_x
decode_bbox_center_y = mbox_loc[:, 1] * prior_height * 0.1
decode_bbox_center_y += prior_center_y
# 真實(shí)框的寬與高的求取
decode_bbox_width = np.exp(mbox_loc[:, 2] * 0.2)
decode_bbox_width *= prior_width
decode_bbox_height = np.exp(mbox_loc[:, 3] * 0.2)
decode_bbox_height *= prior_height
# 獲取真實(shí)框的左上角與右下角
decode_bbox_xmin = decode_bbox_center_x - 0.5 * decode_bbox_width
decode_bbox_ymin = decode_bbox_center_y - 0.5 * decode_bbox_height
decode_bbox_xmax = decode_bbox_center_x + 0.5 * decode_bbox_width
decode_bbox_ymax = decode_bbox_center_y + 0.5 * decode_bbox_height
# 真實(shí)框的左上角與右下角進(jìn)行堆疊
decode_bbox = np.concatenate((decode_bbox_xmin[:, None],
decode_bbox_ymin[:, None],
decode_bbox_xmax[:, None],
decode_bbox_ymax[:, None]), axis=-1)
# 防止超出0與1
decode_bbox = np.minimum(np.maximum(decode_bbox, 0.0), 1.0)
return decode_bbox
def detection_out(self, predictions, mbox_priorbox, background_label_id=0, keep_top_k=200,
confidence_threshold=0.4):
# 網(wǎng)絡(luò)預(yù)測(cè)的結(jié)果
mbox_loc = predictions[0]
# 先驗(yàn)框
mbox_priorbox = mbox_priorbox
# 置信度
mbox_conf = predictions[1]
results = []
# 對(duì)每一個(gè)圖片進(jìn)行處理
for i in range(len(mbox_loc)):
results.append([])
decode_bbox = self.decode_boxes(mbox_loc[i], mbox_priorbox)
for c in range(self.num_classes):
c_confs = mbox_conf[i, :, c]
c_confs_m = c_confs > confidence_threshold
if len(c_confs[c_confs_m]) > 0:
# 取出得分高于confidence_threshold的框
boxes_to_process = decode_bbox[c_confs_m]
confs_to_process = c_confs[c_confs_m]
# 進(jìn)行iou的非極大抑制
feed_dict = {self.boxes: boxes_to_process,
self.scores: confs_to_process}
idx = self.sess.run(self.nms, feed_dict=feed_dict)
# 取出在非極大抑制中效果較好的內(nèi)容
good_boxes = boxes_to_process[idx]
confs = confs_to_process[idx][:, None]
# 將label、置信度、框的位置進(jìn)行堆疊。
labels = c * np.ones((len(idx), 1))
c_pred = np.concatenate((labels, confs, good_boxes),
axis=1)
# 添加進(jìn)result里
results[-1].extend(c_pred)
if len(results[-1]) > 0:
# 按照置信度進(jìn)行排序
results[-1] = np.array(results[-1])
argsort = np.argsort(results[-1][:, 1])[::-1]
results[-1] = results[-1][argsort]
# 選出置信度最大的keep_top_k個(gè)
results[-1] = results[-1][:keep_top_k]
# 獲得，在所有預(yù)測(cè)結(jié)果里面，置信度比較高的框
# 還有，利用先驗(yàn)框和m2det的預(yù)測(cè)結(jié)果，處理獲得了真實(shí)框（預(yù)測(cè)框）的位置
return results
5、在原圖上進(jìn)行繪制
通過(guò)第三步，我們可以獲得預(yù)測(cè)框在原圖上的位置，而且這些預(yù)測(cè)框都是經(jīng)過(guò)篩選的。這些篩選后的框可以直接繪制在圖片上，就可以獲得結(jié)果了。
二、訓(xùn)練部分
1、真實(shí)框的處理
從預(yù)測(cè)部分我們知道，每個(gè)特征層的預(yù)測(cè)結(jié)果，num_priors x 4的卷積 用于預(yù)測(cè) 該特征層上 每一個(gè)網(wǎng)格點(diǎn)上 每一個(gè)先驗(yàn)框的變化情況。
也就是說(shuō)，我們直接利用Efficientdet網(wǎng)絡(luò)預(yù)測(cè)到的結(jié)果，并不是預(yù)測(cè)框在圖片上的真實(shí)位置，需要解碼才能得到真實(shí)位置。
而在訓(xùn)練的時(shí)候，我們需要計(jì)算loss函數(shù)，這個(gè)loss函數(shù)是相對(duì)于Efficientdet網(wǎng)絡(luò)的預(yù)測(cè)結(jié)果的。我們需要把圖片輸入到當(dāng)前的Efficientdet網(wǎng)絡(luò)中，得到預(yù)測(cè)結(jié)果；同時(shí)還需要把真實(shí)框的信息，進(jìn)行編碼，這個(gè)編碼是把真實(shí)框的位置信息格式轉(zhuǎn)化為Efficientdet預(yù)測(cè)結(jié)果的格式信息。
也就是，我們需要找到 每一張用于訓(xùn)練的圖片的每一個(gè)真實(shí)框?qū)?yīng)的先驗(yàn)框，并求出如果想要得到這樣一個(gè)真實(shí)框，我們的預(yù)測(cè)結(jié)果應(yīng)該是怎么樣的。
從預(yù)測(cè)結(jié)果獲得真實(shí)框的過(guò)程被稱(chēng)作解碼，而從真實(shí)框獲得預(yù)測(cè)結(jié)果的過(guò)程就是編碼的過(guò)程。
因此我們只需要將解碼過(guò)程逆過(guò)來(lái)就是編碼過(guò)程了。
實(shí)現(xiàn)代碼如下：
def encode_box(self, box, return_iou=True):
iou = self.iou(box)
encoded_box = np.zeros((self.num_priors, 4 + return_iou))
# 找到每一個(gè)真實(shí)框，重合程度較高的先驗(yàn)框
assign_mask = iou > self.overlap_threshold
if not assign_mask.any():
assign_mask[iou.argmax()] = True
if return_iou:
encoded_box[:, -1][assign_mask] = iou[assign_mask]
# 找到對(duì)應(yīng)的先驗(yàn)框
assigned_priors = self.priors[assign_mask]
# 逆向編碼，將真實(shí)框轉(zhuǎn)化為Efficientdet預(yù)測(cè)結(jié)果的格式
assigned_priors_w = (assigned_priors[:, 2] -
assigned_priors[:, 0])
assigned_priors_h = (assigned_priors[:, 3] -
assigned_priors[:, 1])
encoded_box[:,0][assign_mask] = (box[0] - assigned_priors[:, 0])/assigned_priors_w/0.2
encoded_box[:,1][assign_mask] = (box[1] - assigned_priors[:, 1])/assigned_priors_h/0.2
encoded_box[:,2][assign_mask] = (box[2] - assigned_priors[:, 2])/assigned_priors_w/0.2
encoded_box[:,3][assign_mask] = (box[3] - assigned_priors[:, 3])/assigned_priors_h/0.2
return encoded_box.ravel()
利用上述代碼我們可以獲得，真實(shí)框?qū)?yīng)的所有的iou較大先驗(yàn)框，并計(jì)算了真實(shí)框?qū)?yīng)的所有iou較大的先驗(yàn)框應(yīng)該有的預(yù)測(cè)結(jié)果。
但是由于原始圖片中可能存在多個(gè)真實(shí)框，可能同一個(gè)先驗(yàn)框會(huì)與多個(gè)真實(shí)框重合度較高，我們只取其中與真實(shí)框重合度最高的就可以了。
因此我們還要經(jīng)過(guò)一次篩選，將上述代碼獲得的真實(shí)框?qū)?yīng)的所有的iou較大先驗(yàn)框的預(yù)測(cè)結(jié)果中，iou最大的那個(gè)真實(shí)框篩選出來(lái)。
通過(guò)assign_boxes我們就獲得了，輸入進(jìn)來(lái)的這張圖片，應(yīng)該有的預(yù)測(cè)結(jié)果是什么樣子的。
實(shí)現(xiàn)代碼如下：
def assign_boxes(self, boxes):
assignment = np.zeros((self.num_priors, 4 + 1 + self.num_classes + 1))
assignment[:, 4] = 0.0
assignment[:, -1] = 0.0
if len(boxes) == 0:
return assignment
# 對(duì)每一個(gè)真實(shí)框都進(jìn)行iou計(jì)算
ingored_boxes = np.apply_along_axis(self.ignore_box, 1, boxes[:, :4])
# 取重合程度最大的先驗(yàn)框，并且獲取這個(gè)先驗(yàn)框的index
ingored_boxes = ingored_boxes.reshape(-1, self.num_priors, 1)
# (num_priors)
ignore_iou = ingored_boxes[:, :, 0].max(axis=0)
# (num_priors)
ignore_iou_mask = ignore_iou > 0
assignment[:, 4][ignore_iou_mask] = -1
assignment[:, -1][ignore_iou_mask] = -1
# (n, num_priors, 5)
encoded_boxes = np.apply_along_axis(self.encode_box, 1, boxes[:, :4])
# 每一個(gè)真實(shí)框的編碼后的值，和iou
# (n, num_priors)
encoded_boxes = encoded_boxes.reshape(-1, self.num_priors, 5)
# 取重合程度最大的先驗(yàn)框，并且獲取這個(gè)先驗(yàn)框的index
# (num_priors)
best_iou = encoded_boxes[:, :, -1].max(axis=0)
# (num_priors)
best_iou_idx = encoded_boxes[:, :, -1].argmax(axis=0)
# (num_priors)
best_iou_mask = best_iou > 0
# 某個(gè)先驗(yàn)框它屬于哪個(gè)真實(shí)框
best_iou_idx = best_iou_idx[best_iou_mask]
assign_num = len(best_iou_idx)
# 保留重合程度最大的先驗(yàn)框的應(yīng)該有的預(yù)測(cè)結(jié)果
# 哪些先驗(yàn)框存在真實(shí)框
encoded_boxes = encoded_boxes[:, best_iou_mask, :]
assignment[:, :4][best_iou_mask] = encoded_boxes[best_iou_idx,np.arange(assign_num),:4]
# 4代表為背景的概率，為0
assignment[:, 4][best_iou_mask] = 1
assignment[:, 5:-1][best_iou_mask] = boxes[best_iou_idx, 4:]
assignment[:, -1][best_iou_mask] = 1
# 通過(guò)assign_boxes我們就獲得了，輸入進(jìn)來(lái)的這張圖片，應(yīng)該有的預(yù)測(cè)結(jié)果是什么樣子的
return assignment
focal會(huì)忽略一些重合度相對(duì)較高但是不是非常高的先驗(yàn)框，一般將重合度在0.4-0.5之間的先驗(yàn)框進(jìn)行忽略。
實(shí)現(xiàn)代碼如下：
def ignore_box(self, box):
iou = self.iou(box)
ignored_box = np.zeros((self.num_priors, 1))
# 找到每一個(gè)真實(shí)框，重合程度較高的先驗(yàn)框
assign_mask = (iou > self.ignore_threshold)(iouself.overlap_threshold)
if not assign_mask.any():
assign_mask[iou.argmax()] = True
ignored_box[:, 0][assign_mask] = iou[assign_mask]
return ignored_box.ravel()
2、利用處理完的真實(shí)框與對(duì)應(yīng)圖片的預(yù)測(cè)結(jié)果計(jì)算loss
loss的計(jì)算分為兩個(gè)部分：
1、Smooth Loss：獲取所有正標(biāo)簽的框的預(yù)測(cè)結(jié)果的回歸loss。
2、Focal Loss：獲取所有未被忽略的種類(lèi)的預(yù)測(cè)結(jié)果的交叉熵loss。
由于在Efficientdet的訓(xùn)練過(guò)程中，正負(fù)樣本極其不平衡，即 存在對(duì)應(yīng)真實(shí)框的先驗(yàn)框可能只有若干個(gè)，但是不存在對(duì)應(yīng)真實(shí)框的負(fù)樣本卻有上萬(wàn)個(gè)，這就會(huì)導(dǎo)致負(fù)樣本的loss值極大，因此引入了Focal Loss進(jìn)行正負(fù)樣本的平衡，關(guān)于Focal Loss的介紹可以看這個(gè)博客。
https://blog.csdn.net/weixin_44791964/article/details/102853782
實(shí)現(xiàn)代碼如下：
def focal(alpha=0.25, gamma=2.0):
def _focal(y_true, y_pred):
# y_true [batch_size, num_anchor, num_classes+1]
# y_pred [batch_size, num_anchor, num_classes]
labels = y_true[:, :, :-1]
anchor_state = y_true[:, :, -1] # -1 是需要忽略的, 0 是背景, 1 是存在目標(biāo)
classification = y_pred
# 找出存在目標(biāo)的先驗(yàn)框
indices_for_object = backend.where(keras.backend.equal(anchor_state, 1))
labels_for_object = backend.gather_nd(labels, indices_for_object)
classification_for_object = backend.gather_nd(classification, indices_for_object)
# 計(jì)算每一個(gè)先驗(yàn)框應(yīng)該有的權(quán)重
alpha_factor_for_object = keras.backend.ones_like(labels_for_object) * alpha
alpha_factor_for_object = backend.where(keras.backend.equal(labels_for_object, 1), alpha_factor_for_object, 1 - alpha_factor_for_object)
focal_weight_for_object = backend.where(keras.backend.equal(labels_for_object, 1), 1 - classification_for_object, classification_for_object)
focal_weight_for_object = alpha_factor_for_object * focal_weight_for_object ** gamma
# 將權(quán)重乘上所求得的交叉熵
cls_loss_for_object = focal_weight_for_object * keras.backend.binary_crossentropy(labels_for_object, classification_for_object)
# 找出實(shí)際上為背景的先驗(yàn)框
indices_for_back = backend.where(keras.backend.equal(anchor_state, 0))
labels_for_back = backend.gather_nd(labels, indices_for_back)
classification_for_back = backend.gather_nd(classification, indices_for_back)
# 計(jì)算每一個(gè)先驗(yàn)框應(yīng)該有的權(quán)重
alpha_factor_for_back = keras.backend.ones_like(labels_for_back) * alpha
alpha_factor_for_back = backend.where(keras.backend.equal(labels_for_back, 1), alpha_factor_for_back, 1 - alpha_factor_for_back)
focal_weight_for_back = backend.where(keras.backend.equal(labels_for_back, 1), 1 - classification_for_back, classification_for_back)
focal_weight_for_back = alpha_factor_for_back * focal_weight_for_back ** gamma
# 將權(quán)重乘上所求得的交叉熵
cls_loss_for_back = focal_weight_for_back * keras.backend.binary_crossentropy(labels_for_back, classification_for_back)
# 標(biāo)準(zhǔn)化，實(shí)際上是正樣本的數(shù)量
normalizer = tf.where(keras.backend.equal(anchor_state, 1))
normalizer = keras.backend.cast(keras.backend.shape(normalizer)[0], keras.backend.floatx())
normalizer = keras.backend.maximum(keras.backend.cast_to_floatx(1.0), normalizer)
# 將所獲得的loss除上正樣本的數(shù)量
cls_loss_for_object = keras.backend.sum(cls_loss_for_object)
cls_loss_for_back = keras.backend.sum(cls_loss_for_back)
# 總的loss
loss = (cls_loss_for_object + cls_loss_for_back)/normalizer
return loss
return _focal
def smooth_l1(sigma=3.0):
sigma_squared = sigma ** 2
def _smooth_l1(y_true, y_pred):
# y_true [batch_size, num_anchor, 4+1]
# y_pred [batch_size, num_anchor, 4]
regression = y_pred
regression_target = y_true[:, :, :-1]
anchor_state = y_true[:, :, -1]
# 找到正樣本
indices = tf.where(keras.backend.equal(anchor_state, 1))
regression = tf.gather_nd(regression, indices)
regression_target = tf.gather_nd(regression_target, indices)
# 計(jì)算 smooth L1 loss
# f(x) = 0.5 * (sigma * x)^2 if |x| 1 / sigma / sigma
# |x| - 0.5 / sigma / sigma otherwise
regression_diff = regression - regression_target
regression_diff = keras.backend.abs(regression_diff)
regression_loss = backend.where(
keras.backend.less(regression_diff, 1.0 / sigma_squared),
0.5 * sigma_squared * keras.backend.pow(regression_diff, 2),
regression_diff - 0.5 / sigma_squared
)
normalizer = keras.backend.maximum(1, keras.backend.shape(indices)[0])
normalizer = keras.backend.cast(normalizer, dtype=keras.backend.floatx())
loss = keras.backend.sum(regression_loss) / normalizer
return loss
return _smooth_l1
訓(xùn)練自己的Efficientdet模型
Efficientdet整體的文件夾構(gòu)架如下：
本文使用VOC格式進(jìn)行訓(xùn)練。
訓(xùn)練前將標(biāo)簽文件放在VOCdevkit文件夾下的VOC2007文件夾下的Annotation中。
訓(xùn)練前將圖片文件放在VOCdevkit文件夾下的VOC2007文件夾下的JPEGImages中。
在訓(xùn)練前利用voc2efficientdet.py文件生成對(duì)應(yīng)的txt。
再運(yùn)行根目錄下的voc_annotation.py，運(yùn)行前需要將classes改成你自己的classes。
classes = ["aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "diningtable", "dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor"]
就會(huì)生成對(duì)應(yīng)的2007_train.txt，每一行對(duì)應(yīng)其圖片位置及其真實(shí)框的位置。
在訓(xùn)練前需要修改model_data里面的voc_classes.txt文件，需要將classes改成你自己的classes。
運(yùn)行train.py即可開(kāi)始訓(xùn)練。
修改train.py文件下的phi可以修改efficientdet的版本，訓(xùn)練前注意權(quán)重文件與Efficientdet版本的對(duì)齊。
以上就是Keras搭建Efficientdet目標(biāo)檢測(cè)平臺(tái)的詳細(xì)內(nèi)容，更多關(guān)于Efficientdet目標(biāo)檢測(cè)的資料請(qǐng)關(guān)注腳本之家其它相關(guān)文章！
您可能感興趣的文章:- 利用ImageAI庫(kù)只需幾行python代碼實(shí)現(xiàn)目標(biāo)檢測(cè)
- Python Opencv任意形狀目標(biāo)檢測(cè)并繪制框圖
- 10 行Python 代碼實(shí)現(xiàn) AI 目標(biāo)檢測(cè)技術(shù)【推薦】
- python開(kāi)啟攝像頭以及深度學(xué)習(xí)實(shí)現(xiàn)目標(biāo)檢測(cè)方法
