Q:

What is the difference between test set and validation set?

I found this confusing when I use the neural network toolbox in Matlab.
It divided the raw data set into three parts:

training set
validation set
test set

I notice in many training or learning algorithm, the data is often divided into 2 parts, the training set and the test set.
My questions are:

what is the difference between validation set and test set? 
Is the validation set really specific to neural network? Or it is optional.
To go further, is there a difference between validation and testing in context of machine learning?

A:

Training set: a set of examples used for learning: to fit the parameters of the classifier
 In the Multilayer Perceptron (MLP) case, we would use the training set to find the “optimal” weights with the back-prop rule
Validation set: a set of examples used to tune the parameters of a classifier
 In the MLP case, we would use the validation set to find the “optimal” number of hidden units or
determine a stopping point for the back-propagation algorithm
Test set: a set of examples used only to assess the performance of a fully-trained classifier
 In the MLP case, we would use the test to estimate the error rate after we have chosen the final
model (MLP size and actual weights)
 After assessing the final model on the test set, YOU MUST NOT tune the model any further!
Why separate test and validation sets?
The error rate estimate of the final model on validation data will be biased (smaller than the
true error rate) since the validation set is used to select the final model
After assessing the final model on the test set, YOU MUST NOT tune the model any further!
source : Introduction to Pattern Analysis,Ricardo Gutierrez-OsunaTexas A&M University, Texas A&M University

A:

Typically to perform supervised learning, you need two types of data sets: 

In one dataset (your "gold standard"), you have the input data together with correct/expected output; This dataset is usually duly prepared either by humans or by collecting some data in a semi-automated way. But you must have the expected output for every data row here because you need this for supervised learning.
The data you are going to apply your model to. In many cases, this is the data in which you are interested in the output of your model, and thus you don't have any "expected" output here yet.

While performing machine learning, you do the following:

Training phase: you present your data from your "gold standard" and train your model, by pairing the input with the expected output. 
Validation/Test phase: in order to estimate how well your model has been trained (that is dependent upon the size of your data, the value you would like to predict, input, etc) and to estimate model properties (mean error for numeric predictors, classification errors for classifiers, recall and precision for IR-models etc.)
Application phase: now, you apply your freshly-developed model to the real-world data and get the results. Since you usually don't have any reference value in this type of data (otherwise, why would you need your model?), you can only speculate about the quality of your model output using the results of your validation phase.

The validation phase is often split into two parts:

In the first part, you just look at your models and select the best performing approach using the validation data (=validation)
Then you estimate the accuracy of the selected approach (=test).

Hence the separation to 50/25/25.
In case if you don't need to choose an appropriate model from several rivaling approaches, you can just re-partition your set that you basically have only training set and test set,  without performing the validation of your trained model. I personally partition them 70/30 then.
See also this question.

A:

My 5 years experience in Computer Science taught me that nothing is better than simplicity.
The concept of 'Training/Cross-Validation/Test' Data Sets is as simple as this. When you have a large data set, it's recommended to split it into 3 parts:
++Training set (60% of the original data set): This is used to build up our prediction algorithm. Our algorithm tries to tune itself to the quirks of the training data sets. In this phase we usually create multiple algorithms in order to compare their performances during the Cross-Validation Phase.
++Cross-Validation set (20% of the original data set): This data set is used to compare the performances of the prediction algorithms that were created based on the training set. We choose the algorithm that has the best performance.
++Test set (20% of the original data set): Now we have chosen our preferred prediction algorithm but we don't know yet how it's going to perform on completely unseen real-world data. So, we apply our chosen prediction algorithm on our test set in order to see how it's going to perform so we can have an idea about our algorithm's performance on unseen data.
Notes:
-It's very important to keep in mind that skipping the test phase is not recommended, because the algorithm that performed well during the cross-validation phase doesn't really mean that it's truly the best one, because the algorithms are compared based on the cross-validation set and its quirks and noises...
-During the Test Phase, the purpose is to see how our final model is going to deal in the wild, so in case its performance is very poor we should repeat the whole process starting from the Training Phase.

