Q:

Evaluate performance of Recommendation engine

I have designed item-based recommendation engine with jaccard similarity measure and I incorporated time as well. See the example below:
prod | PR4  | PR5  | time
-------------------------
PR1  | 0.30 | 0.60 | 1
-------------------------
PR2  | 0.60 | 0.50 | 0.3
-------------------------
dot  | 0.48 | 0.45 | - 

On the vertical axis are products that user has, on the horizontal axis those which the user doesn't have. The time indicates when customer bought the product (so PR1 has been bought this month, PR2 3 months before - we assume that products closer to present are more important). And of course, the measure of similarity.
Furthermore, dot product between PR4, PR5 and time is calculated to estimate the best product (this is simplified, there are more steps).
So in this case, I'd offer to this customer product PR4.
Now the struggle I have - How to evaluate the performance of the recommendation engine. Often, (R)MSE is used, but there is no ground truth. And if I remove some products from customer and then observe whether the engine recommends them or not, the prediction will be skewed.
Consider I remove PR2 to test if the engine recommends it. Then PR5 gets recommended in cases where similarity between PR2 and PR1 is less than 0.60.
Which is not necessarily wrong - it only happens because I introduce the engine different information.
This is illustrated below:
prod | PR4  | PR5  | PR2  | time
---------------------------------
PR1  | 0.30 | 0.60 | 0.50 | 1
--------------------------------
dot  | 0.30 | 0.60 | 0.50 | - 

A:

In the end I have adopted the approach in the comments - split into train/test in a way that I randomly choose customers and delete on of their products (train sample) and store it in other dataframe (test sample). Then i can observe whether the product was recommended and calculate MSE for example.
In my case, some customers have only one product, hence I must use for test sample only customers with product count > 1.

