A lot of lolicons just don't care enough to partake in the drama. There's this stereotype of the non-confrontational japanese, I don't think it's super accurate on social medias but does fit with artists ime. Very few artists (lolicon or not) seem to voice their opinion on twitter. I'm not sure whether it's due to their lack of exposure to western dramas, if they don't want to use their artist platform for their opinions or if they are effectively non-confrontational. Similarly, normal lolicons and loli enjoyers don't necessarily voice their opinion either, I never partake in twitter drama for example.
I believe there are a lot more lolicons than antis inside, what could be broadly called, the anime and manga community. Lolis are very old and have been part of anime culture since almost the very beginning and has been the source of one of its big booms. And not just as a niche as most newfags and secondaries like to claim. It's also fair to assume that the number of lolicons increases as anime becomes more popular.
In parallel, the numbers of antis will increase as anime gets more and more exposure to this sort of people. I mean, antis very often coincide with well known cancers from other hobbies and communities. I don't think one should underestimate the role of twitter in this, it gives exposure and voices to people from the "outside", who aren't even interested in the media at all.
I also believe that another problem is the conflation of a lot of concepts and "problems". Like loli has a variety of definitions, the anti loli discussion is very often an anti (lewd) fan service one, etc.