“Fictional art doesn’t harm real kids” can only go so far before something actually bad happens…
It depends on the person, to be honest. I’d like to think most people who enjoy lolicon know the difference between reality and fiction and have self-control. I don’t like it myself but I don’t look down on those who do. Objectively speaking, fictional art doesn’t harm kids, but someone can lead to actual harm, so in a sense, it can indirectly harm kids.