Intro
I got the idea to see how small of a machine learning model could understand American football diagrams (Xs and Os). I have only generated the offensive diagrams (the Os). The interesting things to predict are how many eligible receivers there are and if the formation is valid.
Details
Outputs from this task:
- dataset huggingface
- generator and models github
Some of the diagrams are similar to what you might see in a football game, and others are not -- but both contain invalid and valid diagrams. The diagrams aren't all the possible valid formations.
The valid vs invalid formation is the easier task and relies on counting the number of Os that there are at the line of scrimmage: there should be 7+. The diagrams always put the line of scrimmage at the same vertical location
The number of eligible receivers is a slightly harder task, and relies on if there is another player to the further outside on the line of scrimmage. The two outside on the line of scrimmage players and then every backfield position excluding the quarterback (for this dataset). For this dataset, I made the quarterback red and not a counted eligible receiver. There is a considerable amount of jitter in the placement of players in the diagrams and some of the linemen could be counted as not on the line of scrimmage in a real game.
Models
I tried two different models on the same task. The models had two output heads during training: a binary classification head (was the formation valid or not) and a multi-class classification head (1 - 5 eligible receivers).
The first model was a pure convolutional model common in image classification tasks, MobileNetV2, that I trained for 20 epochs, choosing the model with the best accuracy on the number of eligible receivers task. I trained MobileNetV2 without the pretrained weights as those weights caused the training to be unstable.
The model chosen:
epoch 18/20 train_loss=0.0112 train_formation_acc=0.9994 train_receiver_acc=0.9977 test_loss=0.0008 test_formation_acc=1.0000 test_receiver_acc=0.9996=== formation_invalid (binary) ===
confusion matrix (rows=actual, cols=predicted):
pred_valid pred_invalid
actual_valid 9591 0
actual_invalid 0 409
accuracy: 1.0000
precision: 1.0000
recall: 1.0000
f1: 1.0000
type I error rate (false positive rate, valid called invalid): 0.0000
type II error rate (false negative rate, invalid called valid): 0.0000
=== eligible_receiver_count (multi-class) ===
actual\pred 1 2 3 4 5
1 0 0 0 0 0
2 0 0 0 0 0
3 0 0 704 0 0
4 0 0 1 3625 0
5 0 0 0 3 5667
class precision recall f1 support
1 0.0000 0.0000 0.0000 0
2 0.0000 0.0000 0.0000 0
3 0.9986 1.0000 0.9993 704
4 0.9992 0.9997 0.9994 3626
5 1.0000 0.9995 0.9997 5670The second model has Axial attention -- attention along the width and height of the image separately -- and Convolutions with skip connections. This model is in the train-tinyaxialmobilenet.py file in the Github repo.
The best training epoch:
epoch 17/20 train_loss=0.0050 train_formation_acc=1.0000 train_receiver_acc=0.9986 test_loss=0.0004 test_formation_acc=1.0000 test_receiver_acc=0.9999=== formation_invalid (binary) ===
confusion matrix (rows=actual, cols=predicted):
pred_valid pred_invalid
actual_valid 9591 0
actual_invalid 0 409
accuracy: 1.0000
precision: 1.0000
recall: 1.0000
f1: 1.0000
type I error rate (false positive rate, valid called invalid): 0.0000
type II error rate (false negative rate, invalid called valid): 0.0000
=== eligible_receiver_count (multi-class) ===
actual\pred 1 2 3 4 5
1 0 0 0 0 0
2 0 0 0 0 0
3 0 0 704 0 0
4 0 0 1 3625 0
5 0 0 0 0 5670
class precision recall f1 support
1 0.0000 0.0000 0.0000 0
2 0.0000 0.0000 0.0000 0
3 0.9986 1.0000 0.9993 704
4 1.0000 0.9997 0.9999 3626
5 1.0000 1.0000 1.0000 5670
Conclusion
Both models did well on the task. MobileNetV2 has ~3.5M parameters. The custom model had ~250k parameters, and did better then the pure vision model.
I had thought the problem was more difficult than it actually proved to be. You can find people asking on Reddit and other places whether a formation was valid, and was a receiver eligible, but the actual task as posed in this dataset is easier than I imagined.