If you are saying that you exclusively want to “lock-out” your custom domain only to your specified email delivery server which is Office 365, yes you already did by setting up Sender Policy Framework(SPF). In a nutshell, when you have added Office 365 to your SPF records, the whole internet (all services using SMTP) acknowledge and respect it. That means, each receiving mail server will know that Office 365 is your legitimate email carrier.
Note: SPF(email security mechanism) and SMTP are standard protocols and legitimate Emailing service provider (Google, Yahoo, MSFT.etc) is bound by it.
On the other hand, due to advancement of technology and exploitation, if you exclusively want to “lock-out” or digitally prevent unauthorized use of your domain. That’s not possible specially if the bad actor uses a Email delivery server that is not bound by those standards mentioned above and by default will be deemed illegitimate by the SMTP spectrum.
Bottom-line, email security mechanism always is imperative on the receiving end and should follow standards.