I don't think you would need such a big thing like VGG even (besides, it's inefficient compared to newer architectures). The idea mostly is to extract festures with conv layers and then have an mlp to classify stuff based on the features. An AlexNet like network is the way I always start, just with adapted layer sizes. If it's already enough then no need to slap ResNet on anything.