A couple of things I like.
Firstly, train in the morning before work - you'll still get a better session in on less than optimal sleep than after an 8 hour day of manual labour.
Secondly, get rid of the aspects of your training that have the least impact. Strip it down. What has the best input:output ratio? What has the worst? Get rid of the latter. To paraphrase Carl Johnson, Jonathan Edwards' coach, do the least amount of work necessary to generate improvement. If you can improve training 3 days a week why train 6 times a week if the difference is negligible? If you can still improve training only twice a week then do that. If you can improve training 6 times a week but doing half the work do that. Only you can know what that threshold is, but I'm sure everyone has some garbage training they could ditch.
Also, to borrow from Ivan Abadjiev, if you absolutely cannot rid yourself of the compulsion to train 6 times a week AND do manual labour for 5 then you need to build in progression. You can't just jump in at the deep end. You say at the moment you're working 20 hours a week. Let's say that's split up over 4 days, each 5 hours long. So, you work 20 hours and you train twice in that week. After a month you maybe add another training session, so you're now training 3 times a week and still working 20 hours. You do that for another month, and at the end of that month you add a few more hours of work. In the third month you're still training 3 times a week, but you're now working 24 hours because you added an hour to each day. Do that for a month, and so on. I' sure you can see what I'm getting at. Not that I'd do that.
“Begin at the beginning,” the King said gravely, “and go on till you come to the end. Then stop.”