How typically do validation of the fashions fail?
I am nonetheless fairly new to information science (been engaged on it for over a 12 months)
I labored for a lot of weeks in a posh evaluation (lengthy pipeline with a number of completely different fashions)
The primary downside that we had was that the variety of samples was very low, and therefore, performing valdation methods, corresponding to cross-validation, was discarded.
Now we have now obtained an enormous dataset, (ten instances extra samples that we had), and I used it as a check dataset.
It is true that this new dataset is completely different in lots of facets to the one we used to coach, however you possibly can`t keep away from the reality, the outcomes are completely horrible, even beneath random.
If I had had this originally I may have accomplished one thing significantly better however I really feel now’s too late.
I really feel horrible, my work has no actual worth, how typically does this occur?
Comments ( 2 )
Err. Ok. Why not train your model on your new, big, dataset? If I read correctly it sounds like you had a tiny dataset to train the model, didn’t train it properly or validate, and now it’s not a good fit to new data.. but why would it be?
Also, less than random? That can’t be right. Check the two datasets are labelled in the same way and the data structure is the same etc.
Models always perform terribly if the input data changes significantly. When that happens naturally over time, it’s called data drift; and it’s a very important component of a model’s lifecycle for this very reason. This isn’t quite data drift because the change was more artificial, but the solution is the same: retrain the model on a more representative training set. And make sure the training set is large enough to capture the necessary patterns.