Your surgery report (If you have it) would have details regarding how much of the meniscus was removed. And yes, you can certainly still be causing more damage to those structures in your knee the longer you go on. The meniscus is like a shock absorber, and it gets compressed and ground between the femur and the tibia. If there is move movement at this joint like there is in an ACL deficient knee (That wobblyness/unstable feeling you talk of), then you can imagine that the meniscus is taking more of that 'grinding'.
if you wanted to assess the current integrity of your meniscus, I'm sure you could get an MRI which is your most accurate diagnostic test that isn't invasive.
If your time away from the things you love is what is scaring you, you could consider LARS (Ligament Augmentation and Reconstruction System), which has a much shorter recovery time as it is a synthetic tissue, but there isn't any studies looking at prognosis in the long term with this surgery, which means we don't actually know if there is a greater chance of failure of the graft or of OA.
So again, it comes down to a personal choice. I can totally empathize with you about missing out on such important parts of your life. I would just consider the fact that you have a long life ahead of you, and with healthy joints, you can exercise and play jiu-jitsu for a very long time.
Let me know if you have any other questions