We conducted lateral force microscopy measurements on seven [LaMnO$_3$]$_m$/[SrMnO$_3$]$_n$ superlattices with varied layer thicknesses. We observe that the friction forces and the friction coefficients initially increase with increasing LaMnO$_3$ top layer thickness, followed by saturation when the top layer thickness exceeds a few nanometers. These observations clearly demonstrate that sliding friction is affected by sub-surface material properties to a depth of several nanometers and is not just determined by dynamics in the contact interface. We argue that the sub-surface dissipated energy is governed by damping in the elastically strained volume below the AFM tip, an effect which we estimate via thermoelasticity. The absence of a correlation between friction and the thermal resistivity of our superlattices shows furthermore that high-frequency phonons and heat conduction do not play a role in determining friction. Our observations thus demonstrate that friction can be tailored by sub-surface material properties.