How to compare two neural networks?

There are several metrics:

1. Number of parameters
2. FLOPs
3. GPU occupancy (nvidia-smi)
4. Training/inference time

Here we only talk about FLOPs and parameters. Note: Floating point of operations (FLOPs) is different with floating point of per second (FLOPS). The FLOPS is the same for the same hardware device, but the FLOPs are different for different networks.

FLOPs is related to network design: Number of layers, activation layer selection, parameters, etc.

The difference between FLOPs and parameters is shown at the top figure.

Because Convolution layer can share the kernel, its parameters is far lower than the FLOPs.

PS: Throughput refers to the number of examples (or tokens) that are processed within a specific period of time, e.g., “examples (or tokens) per second”.

PS: Normally MACs (multiply–accumulate operation) are the half of FLOPs.