Carburizing is a critical surface hardening heat treatment applied to steels to enhance their surface hardness, wear resistance, and fatigue performance. The mechanical properties of carburized steels are strongly influenced by microstructural features, particularly the morphology and distribution of carbide phases. The multi-scale morphology of carbides, heterogeneous spatial distribution, coexistence of multiple phases, and variable grain boundary characteristics, together with the inherent noise in metallographic images, pose significant challenges for traditional image analysis methods. This study presents an integrated multimodal learning framework for hardness prediction and carburizing process optimization. The framework combines advanced deep semantic segmentation models for carbide phase detection with machine learning algorithms to predict post-carburization hardness. A high-quality dataset was constructed, covering over 500 entries, involving various steel compositions and 20 different carburizing heat treatment parameters. The best-performing segmentation model, UNet++ with DenseNet121, was used to extract quantitative microstructural features, such as carbide area, perimeter, and uniformity, which were then integrated with alloy composition and process parameters for hardness prediction. Among various machine learning models, MLP (Multi-Layer Perceptron) demonstrated the highest predictive accuracy (R 2 = 0.996, RMSE = 1.95). SHAP (SHapley Additive exPlanations) analysis was employed to identify key features influencing hardness, with chromium content, carbide distribution, and heat treatment parameters emerging as the most critical factors. The framework was validated through experimental data, showing excellent agreement between predicted and measured hardness curves. This work offers a transferable methodology for data-driven optimization in materials design, paving the way for more efficient and precise carburizing process development.