The show experienced its lowest launch since 2006, and viewing figures are down 3 million on last year.
Although the former Pussycat Dolls singer recently replaced Kelly Rowland as the fourth permanent judge, insiders say the team are planning an overhaul of the ITV1 show, which could mean no guest judges in 2013.
A source told The Sun: “The guest judges just don’t work. It’s great for audiences who are watching live but it doesn’t translate well on the telly. People get confused as the lineup’s always changing from song to song.”
In a further shake-up, the show’s launch date may be pushed back to September.
The source said: “It maybe airs a little early and people struggle when they might be on holiday or enjoying the sunshine at home.”
But Cowell has apparently earmarked Mel B for next year’s series according to Now.
“Simon loves the fact she’s upfront and charismatic and she proved herself as a guest judge at this year’s auditions,” said an insider.
“The only reason she couldn’t do it permanently this series was because she’s based in Australia right now and has commitments there. But he wants her for next year’s show.”
Producers hope that Cheryl Cole’s return as Gary Barlow’s guest mentor during the Boot Camp and Judges’ Houses rounds will draw in bigger audiences for the current series.
X Factor continues on Saturday.