Outline
clip_by_value
relu
clip_by_norm
gradient clipping
clip_by_value
import tensorflow as tfa = tf.range(10)
a<tf.Tensor: id=3, shape=(10,), dtype=int32, numpy=array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int32)># a中小于2的元素值为2
tf.maximum(a, 2)<tf.Tensor: id=6, shape=(10,), dtype=int32, numpy=array([2, 2, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int32)># a中大于8的元素值为8
tf.minimum(a, 8)<tf.Tensor: id=9, shape=(10,), dtype=int32, numpy=array([0, 1, 2, 3, 4, 5, 6, 7, 8, 8], dtype=int32)># a中的元素值限制在[2,8]区间内
tf.clip_by_value(a, 2, 8)<tf.Tensor: id=14, shape=(10,), dtype=int32, numpy=array([2, 2, 2, 3, 4, 5, 6, 7, 8, 8], dtype=int32)>relu
a = a - 5
a<tf.Tensor: id=17, shape=(10,), dtype=int32, numpy=array([-5, -4, -3, -2, -1, 0, 1, 2, 3, 4], dtype=int32)>tf.nn.relu(a)<tf.Tensor: id=19, shape=(10,), dtype=int32, numpy=array([0, 0, 0, 0, 0, 0, 1, 2, 3, 4], dtype=int32)>tf.maximum(a, 0)<tf.Tensor: id=22, shape=(10,), dtype=int32, numpy=array([0, 0, 0, 0, 0, 0, 1, 2, 3, 4], dtype=int32)>clip_by_norm
- 缩放时不改变梯度方向
a = tf.random.normal([2, 2], mean=10)
a<tf.Tensor: id=35, shape=(2, 2), dtype=float32, numpy=
array([[ 8.630464, 10.737844],
[ 9.764073, 10.382202]], dtype=float32)>tf.norm(a)<tf.Tensor: id=41, shape=(), dtype=float32, numpy=19.822044># 等比例的放缩a, norm为15
aa = tf.clip_by_norm(a, 15)
aa<tf.Tensor: id=58, shape=(2, 2), dtype=float32, numpy=
array([[6.5309587, 8.125684 ],
[7.388799 , 7.8565574]], dtype=float32)>tf.norm(aa)<tf.Tensor: id=64, shape=(), dtype=float32, numpy=15.0>gradient clipping
Gradient Exploding or vanishing
set lr=1
new_grads,total_norm = tf.clip_by_global_norm(grads,25)
裁剪所有向量，但是所有向量的梯度方向都不变化
内容来源于网络如有侵权请私信删除
- 还没有人评论，欢迎说说您的想法！
客服
