| # |
Apr 23rd 2019, 15:58 |
 josbeir |
yeah ur right |
| # |
Apr 23rd 2019, 15:57 |
dereuromark |
in the end that would also use server resources, wouldnt it? and as such would also need simular knowledge about the task scoring in terms of capacity usage. |
| # |
Apr 23rd 2019, 15:55 |
josbeir |
i would look into amqp (rabbitmq, ..) if resource stuff is getting in the way ? |
| # |
Apr 23rd 2019, 15:53 |
josbeir |
sychronous |
| # |
Apr 23rd 2019, 15:53 |
josbeir |
so each cron is a worker that runs tasks |
| # |
Apr 23rd 2019, 15:52 |
josbeir |
hmm |
| # |
Apr 23rd 2019, 15:51 |
dereuromark |
cronjob based (x minutes until max limit reached per server) |
| # |
Apr 23rd 2019, 15:51 |
josbeir |
never use the plugin |
| # |
Apr 23rd 2019, 15:51 |
josbeir |
(tasks) |
| # |
Apr 23rd 2019, 15:51 |
josbeir |
how does cakephp-queue spawn processes ? |
| # |
Apr 23rd 2019, 15:50 |
josbeir |
rightr |
| # |
Apr 23rd 2019, 15:50 |
dereuromark |
not possible with exec() in php, very hard to find the real values |
| # |
Apr 23rd 2019, 15:49 |
josbeir |
and why not implement resource monitoring of the spawned process ? |
| # |
Apr 23rd 2019, 15:49 |
steinkel |
maybe using `nice` over the worker shells will allow the OS handle that for you |
| # |
Apr 23rd 2019, 15:49 |
dereuromark |
for composer update, yeah. very expensive.. |
| # |
Apr 23rd 2019, 15:49 |
dereuromark |
you fine tune it over time, for now we need a solution to not overload the server(s) :slightly_smiling_face: |
| # |
Apr 23rd 2019, 15:49 |
steinkel |
the problem is CPU consumption making other processes slow in that server? |
| # |
Apr 23rd 2019, 15:48 |
phantomwatson |
Aside from it maybe being difficult to put a number on how "big" a process is, that's a neat solution. If resource use can be measured, then maybe that could be added to logs so people could tweak their "score" values to be more accurate. |
| # |
Apr 23rd 2019, 15:48 |
dereuromark |
this can be even auto-adjustable based on the statistics we already have of finished tasks and their trend. but as a base line a value from 0...100 should already make it clear based on the amount of servers you provide. |
| # |
Apr 23rd 2019, 15:47 |
josbeir |
jobs can get more intensive in time |
| # |
Apr 23rd 2019, 15:47 |
josbeir |
its too abstract |
| # |
Apr 23rd 2019, 15:47 |
josbeir |
@dereuromark i'm not sure a static 'cost' number will be sufficient |
| # |
Apr 23rd 2019, 15:46 |
dereuromark |
jep, and the fetching then allows only to get the jobs that still "fit" the current estimated work load |
| # |
Apr 23rd 2019, 15:46 |
phantomwatson |
@dereuromark, like queued jobs would have a value for how processor-intensive they are, and some maximum simultaneous score threshold is set? |
| # |
Apr 23rd 2019, 15:46 |
dereuromark |
https://github.com/dereuromark/cakephp-queue/issues/226 is my current idea. |
| # |
Apr 23rd 2019, 15:45 |
phantomwatson |
Which didn't preclude the processes running at the same time, it just handled the timing of the rougher parts of those processes. |
| # |
Apr 23rd 2019, 15:45 |
dereuromark |
thats the easiest solution, sure. I was thinking more into a score thing for my queue plugin. |
| # |
Apr 23rd 2019, 15:45 |
josbeir |
the thing is, thats mostly why we have queue workers |
| # |
Apr 23rd 2019, 15:44 |
phantomwatson |
I have an app that runs some very CPU / database intensive tasks and I started running into problems when I'd run more than one of these processes at the same time, so I implemented a cached flag that just says "hold on, another process has called dibs for the moment" that delays all other processes trying to do potentially conflicting / overloading stuff. |
| # |
Apr 23rd 2019, 15:44 |
josbeir |
yeah.. never mind that, there is. |
| # |
Apr 23rd 2019, 15:44 |
josbeir |
there's no such things as bad ideas |
| # |
Apr 23rd 2019, 15:44 |
josbeir |
haha, the modest developer aproach, i hear ya |
| # |
Apr 23rd 2019, 15:42 |
phantomwatson |
I suppose that's just what I tend to say when I anticipate there being a more elegant solution I haven't figured out yet and want to at least look self-aware. :slightly_smiling_face: |
| # |
Apr 23rd 2019, 15:42 |
josbeir |
if it works... |
| # |
Apr 23rd 2019, 15:42 |
josbeir |
@phantomwatson why is it hacky |
| # |
Apr 23rd 2019, 15:41 |
phantomwatson |
Kind of hacky. |
| # |
Apr 23rd 2019, 15:41 |
phantomwatson |
[shrug] A lock flag? |
| # |
Apr 23rd 2019, 15:40 |
josbeir |
wr |
| # |
Apr 23rd 2019, 15:40 |
josbeir |
amqp |
| # |
Apr 23rd 2019, 15:40 |
slackebot |
anything for it in CakeResque, Fresque, Queuesadilla, Gearman either. |
| # |
Apr 23rd 2019, 15:40 |
dereuromark |
How do people do job queuing (those that actually do this kind of thing in their apps^^) when you have limited resources per server and some more expensive tasks? we have some 2GB composer update, and we need to limit simultaneous runs here per queue server (either per job type or by cost of those tasks in general). I have to extend my plugin here, just wondering how the other solutions handle this kind of thing in (Cake)PHP. I didnt see |