Semi-automate before you fully automate

I am currently going through the process of tuning a spam filter on blognami.com. I have an AI classifier that decides if some content is spam or not. I am currently reviewing each classification manually, and then adjusting the classifier if it isn't performing as I want. I am also manually suspending spammers.

It is tempting to go fully automated - so that the suspending of spammers happens as soon as the classifier has made its decision. But experience has taught me that this is a mistake. Automating the wrong thing means you are automating problems - which we don't want. This semi-automated approach gives you a feedback loop that enables your AI solution to earn your trust. Once you trust your AI enough to do (mostly) the right thing you can then turn on full automation.

I think it is good practice to do most things manually first - it allows you to first fully understand the problem, where the real pain is, and based on that what level of automation is really needed - if it is actually needed at all.