Is that intuition, or are you just happy to see me?

Some intellectual wrangling with the Dismal Blogger resulted in us reaching a well-known (atleast to me) impasse. What is the role of intuition and simplicity in science and mathematics, and is it antithetical to rigour and complexity? Despite my ardent desire to learn, I am poorly versed in methodological questions. Therefore, I supply my favourite quotation on this issue. Obviously, being a quotation, it doesn't amount to much, but it is generally how I feel about the rôle of rigour.

The rôles of rigor and intuition are subject to misconceptions. As was pointed out in Volume 1, natural intuition and natural thinking are a poor affair, but they gain strength with the development of mathematical theory. Today's intuition and applications depend on the most sophisticated theories of yesterday. Furthermore, strict theory represents economy of thought rather than luxury.

- An Introduction to Probability Theory and its Applications, Volume II (2^nd edition), Feller, William (1971), pp. 3, footnote 4
In the pantheon of extraordinary textbook writers, Feller is second to none, cramming in as he does relevant example after relevant example, but never giving in to the desire to jump to conclusions based on examples - preferring to work through to confirm/deny his intuition, and using his results to inform his intuition. This last is important, because if like the American astronauts, we were to wait for all problems to be solved rigorously before we ever did statistics, we'd be here a while. The part played by experience and common sense in settings where cost, scale and speed are issues is invaluable. Just yesterday, Andrew Gelman, over on his blog dredged out this quote:

Statisticians are particularly sensitive to default settings, which makes sense considering that statistics is, in many ways, a science based on defaults. What is a "statistical method" if not a recommended default analysis, backed up by some combination of theory and experience?

I can understand that practitioners would typically choose to let the theorists battle it out amongst themselves for a decade or two to figure out relative merits of techniques, and only then move to gather up the spoils of war i.e. the most-widely applicable/median/default method. But I fear this is sub-optimal. There is little comfort in something that often works, but fails when you need it to work most (e.g. macroeconomics).
On a related note, I am little uncomfortable with the phrase - complexity for complexity's sake. There is, in my opinion, no such thing. Complexity relates to how you “do” science. Proofs can be elegant or workmanlike and algorithms & techniques can be crude or sophisticated, and you can choose one or the other based on your preferences. But the actual content of science, theorems, are exactly that, little bits of value-neutral truth which give you the power to discriminate. And just like John Locke:

To love truth for truth's sake is the principal part of human perfection in this world, and the seed-plot of all other virtues.

so we too claim.
Is all rigour a good thing? Obviously not. But useless rigour is easily seen for what it is (if assessed by a trained person) and cast aside. Not that easy to cast aside rules-of-thumb or traditions or defaults, because by definition, these things are hard to pin down.
Further reading

1. Counterexamples in Probability And Statistics
2. Godel's Proof (chapter II)