Variable selection methods and model selection approaches are valuable statistical tools that are indispensable for almost any statistical modeling question. This review first considers the use of information criteria for model selection. Such criteria provide an ordering of the considered models where the best model is selected. Different modeling goals might require different criteria to be used. Next, the effect of including a penalty in the estimation process is discussed. Third, nonparametric estimation is discussed; it contains several aspects of model choice, such as the choice of the estimator to use and the selection of tuning parameters. Fourth, model averaging approaches are reviewed in which estimators from different models are weighted to provide one final estimator. There are several ways to choose the weights, and most of them result in data-driven, hence random, weights. Challenges for inference after model selection and inference for model-averaged estimators are discussed