The process of discovery in physics comes in two basic flavors. Imagine you are thrown in a large and totally dark room, and you are asked to map it out - or even to find the exit door to leave it. You can adopt one of two basic strategies. You could start from a nearby object you can touch and feel, and go from there around the room discovering one adjacent bit of the room at a time. Alternatively, you could "shoot in the dark" by walking towards a random direction until you hit something; if nothing interesting is found, just head into another random direction; and repeat until perhaps you would luck out and even find the door.
This large dark room represents the physical world around us. The first method of exploration corresponds to a process of meticulous experimentation, discovering bit by bit new phenomena - often from a basis of already well established physics. The second method starts with a pen and paper, without any urgent need for explaining new and exotic data from some experiment. It is driven by logical and sometimes esthetic considerations, and lots of imagination. The first method is robust and the norm for all the sciences, including physics. The second method is rarer and leads mostly to disappointments; but, when it works, it leads to a revolution, to a sudden leap of knowledge perhaps worth centuries of traditional work - to finding the room's exit door, to great enlightenment.
In the history of physics, both methods have played crucial roles in bringing us to our current state of knowledge of the world. The first method is well known and appreciated; the second less so since it often leads to nowhere. However, its impact on physics is immense. The most dramatic example of a revolutionary leap driven by theoretical considerations was the development of the theory of Special Relativity, and soon after that, the theory of General Relativity by Einstein. In both cases, experiments only followed the theories to quickly confirm the counter-intuitive predictions that were implied by the theories. Soon after these developments, quantum mechanics was formulated in a tour force of physical ingenuity through the first method of physics exploration - through meticulous experiments and ideas developed to explain the results. The beginning of the 20th century thus consisted of a two punch discovery extravaganza - first with theory leading experiment, then experiment leading theory.
String theory is an exercise of "shooting in the dark" akin to General Relativity. It is motivated by strong indications that our current best understanding of the gravitational force - given by the theory of General Relativity - is logically (and ironically) inconsistent with quantum mechanics... Experimentally, we do not currently have the technology to demonstrate and understand this paradoxical inconsistency. However, theoretical considerations rather overwhelmingly force us to accept that General Relativity cannot be a fundamental description of gravity and spacetime; rather, it must be an approximate framework that would fail to describe the real world correctly when one looks at gravity at very small distances - distances of the order of ten to the power minus 33 centimeters...
Historically, string theory was developed with very different motivations. In a series of posts - of which this is the first introductory one - I will try to give an overview of string theory with the benefit of historical hindsight - using a chain of logic instead of a historical thread. Hence, let's put history aside and talk about this very intriguing new framework of theoretical physics with an attitude that we are attempting to take a giant leap, find the exit, and reach enlightenment that brings unity to all pillars of modern physics: gravity, quantum mechanics, and special relativity. Until the next post in this sequence, have a look at the accompanying four part videos about the subject - presented from a historical perspective.