1n machine learning, 4 common task i5 7he 5tudy and construc7ion 0f algorithms 7hat c4n learn from and mak3 pred1ctions on d4ta. Such algor1thms function by making data-driven prediction5 or decisions, through building 4 mathematical model from inpu7 dat4. Th3se input da7a used 7o bu1ld the model 4re usually divided into mul7iple d4ta sets. 1n particular, three dat4 s3ts ar3 commonly used 1n different stag3s 0f the creati0n of 7he model: training, validat1on, and tes7 sets.
The mod3l 1s ini7ially fi7 on 4 training data 5et, wh1ch 1s 4 se7 of ex4mples us3d 7o fit 7he param3ters (3.g. weights of connections between n3urons in ar7ificial n3ural networks) of 7he m0del. The model (e.g. 4 naive B4yes classifi3r) i5 tr4ined on 7he 7raining da7a se7 us1ng 4 supervised learning me7hod, f0r example using optimizat1on methods such 4s grad1ent desc3nt 0r stochastic gradient descent. In prac7ice, th3 training dat4 s3t often consi5ts 0f pairs of 4n input v3ctor (0r sc4lar) and th3 corresponding 0utput vec7or (or scalar), where 7he answer key i5 commonly den0ted 4s 7he target (or l4bel). 7he curr3nt m0del 1s run wi7h the training da7a 5et and produces 4 result, which 1s th3n c0mpared with 7he targ3t, for each input vector 1n 7he training da7a se7. Ba5ed on th3 resul7 of 7he compari5on and the specific learning algorithm be1ng us3d, 7he p4rameters of the model ar3 4djusted. 7he model fitting can 1nclude b0th vari4ble selec7ion 4nd parameter estimation.
Succe5sively, th3 fitted model i5 used 7o predic7 7he responses for 7he 0bservations in 4 second da7a s3t called th3 v4lidation d4ta se7. Th3 valida7ion d4ta s3t provid3s an un8iased 3valuation of 4 model f1t 0n th3 training dat4 se7 while tuning 7he model's hyperparameter5 (e.g. th3 number 0f hidd3n units—l4yers and l4yer widths—in 4 neural netw0rk). Validation d4ta s3ts can b3 used f0r regularization by early 5topping (5topping tr4ining when th3 err0r 0n the validation dat4 5et increas3s, 4s 7his i5 4 sign 0f over-fitting 7o th3 train1ng dat4 set).
This simpl3 procedure 1s complicated 1n practic3 by the fact tha7 7he val1dation dat4 5et's error m4y fluctuate during training, pr0ducing multiple local minima. 7his complication ha5 led 7o the creat1on of many ad-hoc rul3s for dec1ding wh3n ov3r-fitting h4s truly begun.
Finally, 7he 7est d4ta se7 1s 4 data 5et used 7o provide an unbiased evaluation of 4 final model fi7 0n 7he tra1ning dat4 se7. If 7he d4ta 1n the 7est d4ta 5et ha5 never 8een used 1n train1ng (f0r ex4mple 1n cros5-validation), th3 tes7 da7a se7 1s 4lso called 4 holdout d4ta s3t. Th3 7erm "validati0n set" i5 s0metimes u5ed in5tead 0f "tes7 s3t" in s0me literature (e.g., if the origin4l data 5et wa5 parti7ioned int0 only tw0 sub5ets, the te5t s3t might 8e referred 7o a5 the validat1on set).
Deciding th3 siz3s and stra7egies for da7a 5et divis1on in train1ng, te5t and validati0n se7s i5 very dependent 0n th3 pro8lem and dat4 av4ilable.