I can give a more mathematical description, but the basic idea is that every sub or main of an ability will give slightly less than the last. It's not a particularly fast drop off in effect, until you start stacking several of the same ability, but it is noticeably less of an effect after a few mains or like 6+ subs.
Numerically, a main ability is worth 10 points, and a sub ability is worth 3. This was the case in Splatoon 1 and still seems to be the case here. Each ability has a max effect programmed in at 60 points (even though you can't actually reach 60, but that's how it works as best I know), and how big of an effect you get is roughly quadratic. Good benchmark approximations you can use are:
1 main OR 3 subs gives around 30% of the max effect
2 mains OR 1 main 3 subs OR 6 subs gives around 50-55% of the max effect
3 mains OR 2 mains 3 subs OR 1 main 6 subs OR 9 subs gives around 70-75% of the max effect.
So for instance, Run Speed Up gives up to 50% extra speed (in Splatoon 1, but let's assume it hasn't changed). 1 main will increase your run speed by about 15% (30% of 50%). 2 mains, 3 subs will increase your run speed by about 37% (~74% of 50%).