Magnesium silicate hydroxide (MSH) has attracted much attention as lubricant additive for its environmental-friendly property. However, its hydrophilicity hindered the industrial application in lubricating oils. Here, we used octadecyltrimethoxysilane (ODTMS) to modify freeze-drying magnesium silicate hydroxide (F-MSH) to obtain ODTMS@F-MSH nanomaterial, which functions well in lubricants free of any dispersant. Tribological tests indicate that, compared with a fully formulated oil, the addition of ODTMS@F-MSH reduces wear effectively (∼25.8%). The magnesium, silicon, and oxygen-containing tribo-film formed on the worn surface. The tribo-film forming mechanism was investigated by atomic force microscope (AFM) experiments and Molecular dynamics (MD) simulations. The results indicate that the ODTMS@F-MSH nanoparticles are of much lower stiffness and higher adhesion than ball-milling MSH (B-MSH), thus reducing the possibility of scratching the rubbing surface and promoting nanoparticles’ adsorption on it. Then the solid tribo-film is formed due to sliding contact between moving parts, thereby enhancing the anti-wear performance.