Conversation

@charlotte can you give an example of what you mean?

1
0
0

@deetwenty I have a way to run a script when a build finishes in that CI system I use, and I want to upload the build artifacts. But i don’t want to upload the build artifacts in parallel because that is very slow and will reupload stuff unnecessarily, so i basically want to append “build artifacts 1234” to a queue, and upload those artifacts to the server in order

2
0
0

@erk not what i mean i think but thanks

0
0
0

@deetwenty hm i could probably solve it in the worst way possible, cronjob that runs every 10m and just checks a single file, uploads the artifacts whose name was written to a file

0
0
0

@charlotte
if you mean specific to systemd I have no idea, otherwise GNU parallel might do what you want?

0
0
0

@charlotte is uploading via rsync a possibility? That should at least be able to prevent reuploads, also you might want to check the network, in general parallel uploads should be faster

1
0
0

@deetwenty no rsync is not a possibility here, no parallel uploads are already done, plus i’m bandwidth limited to 4Mbps so it’s not actually faster, the issue is that if two builds share a dependency that isn’t uploaded yet, if these are uploaded separately in two processes the dependency would be uploaded multiple times. if one was uploaded after the other, or both in the same process that would not happen

1
0
0

@charlotte ah, yikes, yeah that is a problem that isn't easy to solve (especially without something like rsync)

1
0
0

@deetwenty also rsync probably has the same issues? If you try to upload the same file simultaneously

1
0
0

@charlotte with rsync you would first build to a local place that has a similar file structure layout as the desired one on the other end point and then push that full file tree, rsync would then take care of any duplicates. In general I would try to stay away from a "build partly, upload, build some more parts, upload, etc" way of working

1
0
0

@deetwenty with nix it’s an all-or-nothing kinda thing anyways. I managed to write something like a queue though

0
0
0