Q:

What exactly is an instance in Java?

What is the difference between an object, instance, and reference? They say that they have to create an instance to their application? What does that mean?

A:

An object and an instance are the same thing.
Personally I prefer to use the word "instance" when referring to a specific object of a specific type, for example "an instance of type Foo". But when talking about objects in general I would say "objects" rather than "instances". 
A reference either refers to a specific object or else it can be a null reference.

They say that they have to create an instance to their application. What does it mean?

They probably mean you have to write something like this:
Foo foo = new Foo();

If you are unsure what type you should instantiate you should contact the developers of the application and ask for a more complete example.

A:

"instance to an application" means nothing.
"object" and "instance" are the same thing. There is a "class" that defines structure, and instances of that class (obtained with new ClassName()). For example there is the class Car, and there are instance with different properties like mileage, max speed, horse-power, brand, etc.
Reference is, in the Java context, a variable* - it is something pointing to an object/instance. For example, String s = null; - s is a reference, that currently references no instance, but can reference an instance of the String class.
*Jon Skeet made a note about the difference between a variable and a reference. See his comment. It is an important distinction about how Java works when you invoke a method - pass-by-value. 

The value of s is a reference. It's very important to distinguish between variables and values, and objects and references.

A:

When you use the keyword new for example JFrame j = new JFrame(); you are creating an instance of the class JFrame.   

The new operator instantiates a
  class by allocating memory for a new
  object and returning a reference to
  that memory.
Note: The phrase "instantiating a class" means the same thing as
  "creating an object." When you create
  an object, you are creating an
  "instance" of a class, therefore
  "instantiating" a class.

Take a look here
Creating Objects

The types of the Java programming
  language are divided into two
  categories: primitive types and
  reference types.
  The reference types
  are class types, interface types, and
  array types.
  There is also a special
  null type.
  An object is a
  dynamically created instance of a
  class type or a dynamically created
  array.
  The values of a reference
  type are references to objects.

Refer Types, Values, and Variables for more information 

