Q:

How does simulation software work at a fundamental level?

There are lots of software packages like Schr­­­ödinger Maestro or Argus Lab, which are used for simulations of docking.
I have always been fascinated as to how such software works. What is the fundamental principle working "under the hood" of this kind of sofware? 
Let's say I wanted to make a simple piece of software which, when I input the atoms and atom number (e.g., C1H4) can be used for finding different energy levels (or other properties) in a molecule such as methane ($\small\ce{CH4}$). 
Where does one have to start in attempting to make such a package, given a considerable knowledge of programming? What theory is applicable?

A:

Well, that's a rather diffuse question. From what I have seen, the software packages you mention do Molecular Mechanics (MM), which is a treatment of molecules with classical physics. For each atom in a given molecule, the forces acting upon it due to chemical bonding with other atoms is calculated using a set of empirically determined parameters (a force field), and then an estimate of the potential energy is provided. But given that this question is tagged with "Computational Chemistry", I also want to mention quantum chemistry, i.e. the application of quantum mechanics to atoms and molecules. This field is both far more interesting and far more difficult than the classical MM treatment.
A first shot would be the Wikipedia article about Computational Chemistry which already points you to other relevant topics, i.e. the article about Molecular Mechanics, which is quite concise about the topic (although I don't know how exhaustive it is in this regard, since I don't deal with MM in my research). For quantum chemistry, the necessary knowledge is more spread-out; relevant Wikipedia articles would be Eigenproblems, the Schrödinger equation and Ab initio methods. I don't know any literature about MM, but if you want to look deeper into quantum chemistry, I would recommend the following two books (and lots and lots of time and patience for reading them):

Attila Szabo, Neil Ostlund: "Modern Quantum Chemistry". Provides a good mathematical introduction and thorough treatments of the Hartree-Fock and Moller-Plesset methods. It's cheap, also.
Ira Levine: "Quantum Chemistry". Covers a wider range of methods, but is in my opinion a bit more difficult to consume than Szabo/Ostlund.

