The Valley is defined by the Santa Cruz Mountains on the West, and the Diablo Range on the East. San Francisco is often considered part of Silicon Valley, but geographically it is not; but also historically it played only a negligible role in the development of Silicon Valley.
The vast majority of the original companies that defined Silicon Valley were started from San Mateo and south. This area was mostly farm land until post-war, which then steadily converted to suburbs and office parks. Due to the large area of available land,and earthquake concerns, the majority of buildings are just 2-3 stories. Even downtown San Jose, which has the tallest buildings is limited in how tall primarily to its proximity to its own airport.