Make affinity module optional

This was overlooked in the review, sorry.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>