I want to share some of the insights from development of our main (and for now only) model.
MapAssistant v5 is obviously a 5th version of MapAssistant, but it was not always a ML model. Firstly it was an excel file that was using stats returned by FaceitForecast and just multiplied them together: Maps Played * Win Rate * Avg Kills.
It is stupidly obvious that it was simple, but it gave me idea to start development for real with question: „Can I predict which map is the best to play against particular opponents with particular teammates?”. It was also inspired by few stupid teammates that I encountered that are FACEIT level 8 and have no idea that Nuke exist. Obviously they had no idea how to play it either.
I won’t talk about the development of v1-v4, these are for other post, for this I want to focus on v5.
Departure from v4
v4 models were first that gave any results, but their accuracy was not satisfactory:
This result was marginally better than flipping a coin, so not a great success.
The main issue with v4 was it’s db that was constantly deleted and redone because I didn’t capture some important stat. So v5 was mainly just v4, but with manageable db.
So as you may expect the first v5 model was as good as last v4 model scoring 56% accuracy, but with better database and easier features management quality quickly rose:
The setback
As you can see on the graph first 5 versions of v5 were a great success, but v5.6-5.10 were quite disappointing. I was engineering new features (and adding ones that were obvious but I didn’t include them at the beginning because of my dumbness) with no effect. I was also gathering more and more data:
This one presents how many games I was training model on and what little improvement it gave and actually when I gathered advertised 50k matches the model regressed to 65.3% accuracy (sue me). It gave me impression that we maybe reached the limit of accuracy of prediction of something unpredictable. So I tried something new, I built new feature set from ground up. I removed from it useless stats like MVPs and made every stat dependent on something, for example i removed kills and added kills pre round. And this was a catastrophic disaster:
When moving to the new architecture the accuracy got destroyed and upon investigating I found out that my stupid scraper was not filling in Faceit Level columns as well as for some matches all stats columns were empty. With v11 feature set more dependent on those stats it is clear why it got decimated.
Future plans
Now I am working on MapAssistant v5.16, this will be first multimodal version of MapAssistant, including signal from two new specialised models:
I hope the v5.16 lands this week, but for this time I hope that v5.10b is good enough so you won’t blame it when you lose a good map according to MapAssistant.
See you next time.
Put the findings to work.
Open MapAssistant