Harmony search algorithm, which simulates the musical improvisation process in seeking agreeable harmony, is a population based meta-heuristics algorithm for solving optimization problems. Although it has been successfully applied on various optimization problems; it suffers the slow convergence problem, which greatly hinders its applicability for getting good quality solution. Therefore, in this work, we propose a hybrid meta-heuristic algorithm that hybridizes a harmony search with simulated annealing for the purpose of improving the performance of harmony search algorithm. Harmony search algorithm is used to explore the search spaces. Whilst, simulated annealing algorithm is used inside the harmony search algorithm to exploit the search space and further improve the solutions that are generated by harmony search algorithm. The performance of the proposed algorithm is tested using the Solomon's Vehicle Routing Problem with Time Windows (VRPTW) benchmark. Numerical results demonstrate that the hybrid approach is better than the harmony search without simulated annealing and the hybrid also proves itself to be more competent (if not better on some instances) when compared to other approaches in the literature.