Hey Andrew that's a great question and I'm sure one that lots of Cycle users have thought about.
The answer really is... it depends. But let's take a deeper look at what it depends on.
The first step here is to identify the needs of the application. For smaller applications, background services, and the like, it's ok to just run one service per environment. In these cases, the optimizations from putting services on each of the servers or moving the services onto the same server shouldn't be noticeable.
Now if you have an application that has deep network latency requirements, then yes you could start putting a service in each datacenter you're deploying containers to for that environment. Putting one on every server, on the other hand, would be overkill.
And in the middle of these two is a pretty wide range of needs that you can fill by making changes and seeing what the impact is on your service. If you feel like you'd like to really optimize for latency, try a few different strategies and check the logs to see what seems to be working best.
(Great news here as well, more logging/graphs/monitoring is coming each month!)
Side Note
There are some languages, like nodeJS, that will create a DNS lookup for every request. This could really clog the discovery service if things get out of hand, so if that's a concern for you, increasing the number of discovery services is something that could help.
I'd love to hear more about what you end up trying out and what works for you!
