Ningx - limit_rate (download speed) for s2member served files

How can I limit the download rate for files served with s2member?

location ~* /(s2member-files) {
deny all;
gzip off;
limit_rate_after 1m;
limit_conn perip 4;
limit_rate 5000k;
access_log off; log_not_found off;

Obviously does not work - as s2member is serving the files outside the normal rules. I know I could restrict the whole server block to a limit_rate - but that would also apply to the website in general which I do not want to… (I essentially want to allow a maximum of 4 concurrent connections to the s2member-files folder, each at a maximum speed of 5000KB/s - however I obviously do not want to limit the webserver in general to 4 connections per IP only).

I need to do some limiting as sometimes during the evening some people download files very fast while for others the speed is becoming too slow. I have many downloads served by s2member that are over 5GB big (and traffic can be up to 200GB / hour which is creating problems on 1Gbit/s connection and the big files stored on HDD and not on NVME like the rest of the website).

Now the problem is if I just do a limit_rate 6000k; this limit is pretty much useless if someone uses a download accelerator that opens up 20 connections (even better would be of course if there was a way to limit all combined connections per IP to 100Mbit/s for me - irrespective how many connections someone is opening).