INDIANAPOLIS (AP) - Jim Irsay has big plans for the Colts' future. He just hasn't worked through the details yet.

Two weeks after overhauling the front office, Indianapolis took the next big step in its major housecleaning project Tuesday - firing coach Jim Caldwell after three seasons.

This (search) is something that's going to start immediately and I really think we're going to get a coach that's going to lead us going into the future, and I think it's a bright future,'' Irsay said Tuesday.It's tough to change and go forward, we've had such excellence and greatness here over such a long period of time and that's what I expect us to do again.''

Irsay is following the same plan he installed after the 1997 season. Back then, a season-ending loss on the road allowed the 3-13 Colts to clinch the No. 1 overall draft pick. The next day, Irsay fired the general manager and the head coach and eventually changed quarterbacks.

It could happen again with a little more deliberation.

The day after losing at Jacksonville to post their worst record (2-14) in two decades and wrap up this year's No. 1 pick, Irsay fired team vice chairman Bill Polian and his son, Chris, the Colts' general manager. Last week, Irsay hired 39-year-old Ryan Grigson as the replacement for the Polians.

Since then, Grigson and Irsay have been in almost constant meetings debating the future of the coaching staff.

On Monday, Caldwell and former Rams coach Steve Spagnuolo met in Indianapolis about the Colts' defensive coordinator job. Most took it as an indication Caldwell would be retained. That was still the conventional wisdom Tuesday.

Eventually, Irsay and Grigson proved the pundits wrong. Irsay said he told Caldwell of the decision at about 2 p.m., shortly before the team confirmed the firing.

We've talked about where we want to be more balanced,'' Irsay said.We want to be excellent on defense and more consistent, and I think that's something that we're looking at as part of the vision. I don't think the guy has to be offensive or defensive or anything like that. It's a heavy lifting process right now.''

It's unclear where the Colts will turn next.

Yes, Grigson acknowledged, he has a short list of candidates. Not surprisingly, he didn't say who was on the list, which could include names such as Packers offensive coordinator Joe Philbin, Eagles offensive coordinator Mary Mornhinweg and perhaps Spagnuolo.

Neither Grigson nor Irsay provided a timeline for the hire.

We want leadership. Leadership is important,'' Grigson said after making his first big decision in charge of an NFL team.We want strong leadership, and we want someone who shares his vision in this new era of Colts football. We want the best man and the best leader and the man that gives us the best way to go.''

One thing they did agree on: The future didn't include Caldwell

After winning his first 14 games, an NFL record for a rookie head coach, and becoming only the fifth first-year coach to take his team to the Super Bowl, Caldwell did a masterful job guiding the injury-plagued Colts through a thicket of injuries and back to the playoffs in 2010.

But those successes all came with Peyton Manning, who led the Colts to a league-record 115 regular-season wins in the previous decade and a record-tying nine straight playoff appearances.

This year, with Manning out the entire season, the Colts lost their first 13 games.

Among players and coaches, Caldwell was universally well-liked. The list included Manning, who won all four of his record-setting MVP Awards with Caldwell on Indy's staff, as well as perennial Pro Bowl center Jeff Saturday.

I think coach Caldwell has done a very good job. He has gotten the most out of his players, and we play hard for him each and every week,'' Saturday said before the season finale.We haven't necessarily played well, we've made mistakes and done things, but they have, oftentimes, been things that we've talked about in coaching meetings.''

Outsiders often saw it another way.

Fans frequently complained about Caldwell's game management, and some critics referred to Caldwell as a ``puppet'' of the Polians.

Many never forgave Caldwell for pulling the plug on a perfect season in a Game 15 loss to the Jets in 2009 and pointed to the midseason firing of defensive coordinator Larry Coyer and the long delay in replacing Curtis Painter with Dan Orlvosky at quarterback as decisions that should have come much earlier.

Irsay and Grigson did not characterize Caldwell's 1,099-day tenure the same way as fans.

But with Grigson already searching for a new coach and presumably preparing to take Andrew Luck with the No. 1 pick, the questions now turn to Manning, who had his third neck surgery in 19 months on Sept. 8.

The Colts still are not saying much about Manning's recovery, and Grigson has not yet spoken with Manning, who is owed a $28 million bonus in early March.

We're not even there with anything regarding Peyton Manning just yet,'' Grigson said.We have to know about his medical stuff, first. There's so many things there.''

Caldwell ended his Colts' tenure 26-22 overall with one AFC title, two division crowns and one bleak season that has left him unemployed just three years after replacing close friend Tony Dungy, the first black coach to hoist the Lombardi Trophy.

This was a difficult decision,'' Irsay said.I wanted to make sure we took all the time we needed to make sure it was the right decision. ... And just like 14 years, ago, it's a big change for the franchise and at the same time, there's players, coaches, many people on the staff that will go into the new day and get on with the work of 2012.''