Dish (Daniël’s Swish-like activation, 2022) is an activation function that has a shape similar to GELU, Swish and Mish. However, it does not use any elementary functions like exp or erf, making it much faster to compute on most hardware. The function is:
which uses the sigmoidal function .
Sample implementation
Here is a loop that applies Dish to a float array in RISC-V assembly using the Vector extension (RVV), with the reciprocal square root estimation (vfrsqrt7.v) and one additional iteration of the Newton–Raphson method:
11a7a: 0d35f7d7 vsetvli a5,a1,e32,m8,ta,ma
11a7e: 02056807 vle32.v v16,(a0)
11a82: 00279713 slli a4,a5,0x2
11a86: 8d9d sub a1,a1,a5
11a88: 953a add a0,a0,a4
11a8a: 93081c57 vfmul.vv v24,v16,v16
11a8e: 03875c57 vfadd.vf v24,v24,fa4
11a92: 4f821457 vfrsqrt7.v v8,v24
11a96: 92841057 vfmul.vv v0,v8,v8
11a9a: 93801c57 vfmul.vv v24,v24,v0
11a9e: 9387dc57 vfmul.vf v24,v24,fa5
11aa2: 9f86dc57 vfrsub.vf v24,v24,fa3
11aa6: 928c1457 vfmul.vv v8,v8,v24
11aaa: 93041457 vfmul.vv v8,v16,v8
11aae: 02875457 vfadd.vf v8,v8,fa4
11ab2: 9287d457 vfmul.vf v8,v8,fa5
11ab6: 93041857 vfmul.vv v16,v16,v8
11aba: 02066827 vse32.v v16,(a2)
11abe: 963a add a2,a2,a4
11ac0: fdcd bnez a1,11a7a <dish_vectorized(float const*, unsigned long, float*)+0x22>