Excellent mechanical properties are indispensable for thermal barrier coatings (TBCs), including high hardness and low modulus, as well as excellent creep and wear resistance. This study focuses on nanomechanical properties of RENbO4 (RE = La, Gd, Dy, Y, Yb), and the effects of microstructures on their properties are analyzed. The evolutions of hardness and modulus of RENbO4 are studied based on the changes of RE3+ ionic radius, and the shorter bond lengths lead to the higher hardness and modulus due to the increasing bonding strength. The existence of ferroelastic domains causes the presence of pop out during nanoindentation, which affects the elastic and plastic work. Based on the load-displacement curves, we further analysis the creep behavior and wear resistance of RENbO4 ceramics, and the results show that these two parameters show a negative correlation. Among the studied RENbO4 ceramics, YbNbO4 exhibits the best mechanical properties, and the used methods in current paper can be employed to screen materials with outstanding properties for various applications.