I think everyone has different thoughts on what worked and what didn’t, hence why every change, either with a new idea, or back to an old idea results in falling viewing figures.
Nobody could ever get a consensus on whether silly joke acts had a place in the live shows for example.
Also should bad singers feature in the auditions for people to laugh at, or should it all be nice and positive (and therefore often dull).
Early X Factor was must-watch TV because it had that mix of undiscovered talent and the delusional acts where everyone had a good laugh at their expense. In the live shows, there was always the knowledge that Simon might randomly take against a singer and rip shreds off them in front of the nation at any moment.
Then a few things happened. Some of the more ridiculous auditions started to be obviously put on for the camera, and this was carried through to the live shows where of course it's nowhere near as funny if the act is in on the joke. At the same time criticism started to grow that it wasn't fair to treat people in such a way that they were being mocked, and concerns were raised about the possible effects on mental health of participants, and so they toned it down.
The fundamental problem with the show is that the producers never then found a way to develop that original format without resorting to those tactics, because it was always so reliant on them. Hence we got audience then no audience, sing off then no sing off, chairs then no chairs (then chairs but watered down so that it makes no sense anyway...) and so on it goes.
ITV really should have cut their losses a long time ago and switched to a different format e.g. to The Voice when the BBC originally took it. But of course Cowell didn't want to let it go, and ITV didn't want to upset Cowell and so here we are. Surely even he must now realise it's time to call it a day.