| # |
Apr 23rd 2019, 17:56 |
 challgren |
#226? |
| # |
Apr 23rd 2019, 17:55 |
dereuromark |
;) cool. check out my plan on cost per type |
| # |
Apr 23rd 2019, 17:52 |
challgren |
@dereuromark what I did for CPU intense jobs was group them together and then run that group worker on a separate machine and/or set higher thresholds. I had one queuetask that would download a 98mb csv file and parse it into cake, cake 2 it took over 2 hours to complete, cake 3 it would take 5 minutes (my guess is due to entity patching and not blindly updating the database with every save operation |
| # |
Apr 23rd 2019, 17:49 |
challgren |
@neon1024 I do have a plugin for you https://github.com/bcrowe/cakephp-encrypted-type |
| # |
Apr 23rd 2019, 17:25 |
picasso |
im curious what the process is for a new release of phinx to be created? this particular PR is something i need (double precision support for postgres) and has been in cakephp:master for a couple months, but the latest phinx release (0.10.6) is much older than that. https://github.com/cakephp/phinx/pull/1493 |
| # |
Apr 23rd 2019, 15:58 |
josbeir |
yeah ur right |
| # |
Apr 23rd 2019, 15:57 |
dereuromark |
in the end that would also use server resources, wouldnt it? and as such would also need simular knowledge about the task scoring in terms of capacity usage. |
| # |
Apr 23rd 2019, 15:55 |
josbeir |
i would look into amqp (rabbitmq, ..) if resource stuff is getting in the way ? |
| # |
Apr 23rd 2019, 15:53 |
josbeir |
sychronous |
| # |
Apr 23rd 2019, 15:53 |
josbeir |
so each cron is a worker that runs tasks |
| # |
Apr 23rd 2019, 15:52 |
josbeir |
hmm |
| # |
Apr 23rd 2019, 15:51 |
dereuromark |
cronjob based (x minutes until max limit reached per server) |
| # |
Apr 23rd 2019, 15:51 |
josbeir |
never use the plugin |
| # |
Apr 23rd 2019, 15:51 |
josbeir |
(tasks) |
| # |
Apr 23rd 2019, 15:51 |
josbeir |
how does cakephp-queue spawn processes ? |
| # |
Apr 23rd 2019, 15:50 |
josbeir |
rightr |
| # |
Apr 23rd 2019, 15:50 |
dereuromark |
not possible with exec() in php, very hard to find the real values |
| # |
Apr 23rd 2019, 15:49 |
josbeir |
and why not implement resource monitoring of the spawned process ? |
| # |
Apr 23rd 2019, 15:49 |
steinkel |
maybe using `nice` over the worker shells will allow the OS handle that for you |
| # |
Apr 23rd 2019, 15:49 |
dereuromark |
for composer update, yeah. very expensive.. |
| # |
Apr 23rd 2019, 15:49 |
dereuromark |
you fine tune it over time, for now we need a solution to not overload the server(s) :slightly_smiling_face: |
| # |
Apr 23rd 2019, 15:49 |
steinkel |
the problem is CPU consumption making other processes slow in that server? |
| # |
Apr 23rd 2019, 15:48 |
phantomwatson |
Aside from it maybe being difficult to put a number on how "big" a process is, that's a neat solution. If resource use can be measured, then maybe that could be added to logs so people could tweak their "score" values to be more accurate. |
| # |
Apr 23rd 2019, 15:48 |
dereuromark |
this can be even auto-adjustable based on the statistics we already have of finished tasks and their trend. but as a base line a value from 0...100 should already make it clear based on the amount of servers you provide. |
| # |
Apr 23rd 2019, 15:47 |
josbeir |
jobs can get more intensive in time |
| # |
Apr 23rd 2019, 15:47 |
josbeir |
its too abstract |
| # |
Apr 23rd 2019, 15:47 |
josbeir |
@dereuromark i'm not sure a static 'cost' number will be sufficient |
| # |
Apr 23rd 2019, 15:46 |
dereuromark |
jep, and the fetching then allows only to get the jobs that still "fit" the current estimated work load |
| # |
Apr 23rd 2019, 15:46 |
phantomwatson |
@dereuromark, like queued jobs would have a value for how processor-intensive they are, and some maximum simultaneous score threshold is set? |
| # |
Apr 23rd 2019, 15:46 |
dereuromark |
https://github.com/dereuromark/cakephp-queue/issues/226 is my current idea. |
| # |
Apr 23rd 2019, 15:45 |
phantomwatson |
Which didn't preclude the processes running at the same time, it just handled the timing of the rougher parts of those processes. |
| # |
Apr 23rd 2019, 15:45 |
dereuromark |
thats the easiest solution, sure. I was thinking more into a score thing for my queue plugin. |
| # |
Apr 23rd 2019, 15:45 |
josbeir |
the thing is, thats mostly why we have queue workers |
| # |
Apr 23rd 2019, 15:44 |
phantomwatson |
I have an app that runs some very CPU / database intensive tasks and I started running into problems when I'd run more than one of these processes at the same time, so I implemented a cached flag that just says "hold on, another process has called dibs for the moment" that delays all other processes trying to do potentially conflicting / overloading stuff. |
| # |
Apr 23rd 2019, 15:44 |
josbeir |
yeah.. never mind that, there is. |
| # |
Apr 23rd 2019, 15:44 |
josbeir |
there's no such things as bad ideas |
| # |
Apr 23rd 2019, 15:44 |
josbeir |
haha, the modest developer aproach, i hear ya |
| # |
Apr 23rd 2019, 15:42 |
phantomwatson |
I suppose that's just what I tend to say when I anticipate there being a more elegant solution I haven't figured out yet and want to at least look self-aware. :slightly_smiling_face: |
| # |
Apr 23rd 2019, 15:42 |
josbeir |
if it works... |
| # |
Apr 23rd 2019, 15:42 |
josbeir |
@phantomwatson why is it hacky |
| # |
Apr 23rd 2019, 15:41 |
phantomwatson |
Kind of hacky. |