Project

General

Profile

Actions

Action Item #801

closed

Setup git repo mgmt (re: Fork GTK2? Cairo? omxplayer?)

Added by Hammel about 4 years ago. Updated 4 days ago.

Status:
Closed
Priority:
Immediate
Assignee:
Category:
00 - Basic Build Issues
Target version:
Start date:
18 Dec 2020
Due date:
% Done:

100%

Estimated time:
Severity:
01 - Critical

Description

GTK2 is EOL, apparently. GTK3 is overkill and now it's being replaced by GTK4.

I chose GTK2 because of it's simplicity and lightweight nature. I don't need much in the way of UI toolkits. It might be worth while to fork GTK2 so I always have it available.
The same may be true of Cairo, though that's less likely to go away.

If I'm going to fork, I should do it soon. And it should include the entire GTK2 stack: gobject, glib, gdk, etc. Plus I need to make sure I can generate the docs for all of them.

If I get to OpenGL with Cairo I may not need GTK2 as much but I'd still like it around as a lightweight toolkit with an API to using things like Cairo and/or OpenGL.

omxplayer is being deprecated in favor of vlc. However, vlc is bloated and not very Unix-like (small programs piped together to get bigger things). There are some deprecated dependencies for omxplayer too so if I fork omxplayer I may have to fork them too (unless I can always get them from Buildroot).

Source Documentation
Actions #1

Updated by Hammel about 4 years ago

  • Category set to 00 - Basic Build Issues
  • Assignee set to Hammel
  • Priority changed from Normal to Urgent
  • Target version set to 3.0 - Corrino
  • Severity changed from 03 - Medium to 01 - Critical
Actions #2

Updated by Hammel about 4 years ago

  • Subject changed from Fork GTK2? Cairo? to Fork GTK2? Cairo? omxplayer?
  • Description updated (diff)
Actions #3

Updated by Hammel almost 4 years ago

  • Severity changed from 01 - Critical to 04 - Low
Actions #4

Updated by Hammel about 3 years ago

  • Description updated (diff)
Actions #5

Updated by Hammel over 1 year ago

  • Priority changed from Urgent to Immediate
  • Severity changed from 04 - Low to 03 - Medium
Actions #6

Updated by Hammel over 1 year ago

I need a personal git server before I can do this, since the public ones seem to be pushing their commercial features.

I can try GitLab but it needs to run alongside Apache.

I have setup GitWeb at home, but it's not very user friendly.

There are other options. Looks like maybe Gitea might be one to try.
  • gogs looks much simpler to install and uses a non-standard port that makes it easy to get to without setting up DNS.
Actions #7

Updated by Hammel over 1 year ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 10
Actions #8

Updated by Hammel over 1 year ago

  • Subject changed from Fork GTK2? Cairo? omxplayer? to Setup git repo mgmt (re: Fork GTK2? Cairo? omxplayer?)
Actions #9

Updated by Hammel over 1 year ago

Manual migration from GitLab was tested with xnotesng. It worked perfectly.

The migration was run while my local archives of all repos was symlinked to the gogs repository directory. What gogs did was create my username under that repository directory, then migrated the GitLab repo under that as xnotesng.git. It also added xnotesng.wiki.git as a directory next to xnotesng.git.

I probably don't want the wiki feature, since Redmine has a wiki and I also use pmwiki. So new migrations shouldn't have the wiki feature enabled, however there is no option to disable this when doing the migration. You can switch to an external wiki in the per-repository Settings page but the wiki directory does not get removed. Note, also, that using the builtin wiki fails - there probably needs to be some other software installed to use it. There is no option to set the default to "external wiki", unfortunately.

Actions #10

Updated by Hammel over 1 year ago

One thing to consider: I can mirror the dependent repos (GTK2, Cairo, omxplayer, etc.) into Gogs without also migrating all the PiBox repos. That should be the first step. Then see if I can use those in the build. It appears that mirror's are automatically pulled from upstream by Gogs.

I mirrored omxplayer. It worked except I didn't use the Authentication fields (I thought it would just use the https URL I provided) so the terminal where I started gogs was waiting for me to enter my github credentials. Once I did that it completed successfully. So the tip: use the Authentication fields when mirroring.

Actions #11

Updated by Hammel over 1 year ago

  • % Done changed from 10 to 30

Updated migration script works fine using tokens (not passwords). I found that I should create "orgs" in place of GitLab groups when migrating, but I need to create the groups manually. Alternatively, I could check the GOGS API to see if I can add the group creation with the script.

One other thing I need to check the API for is Mirroring. I want to mirror my repos for now, but convert to regular repositories if GitLab goes wonky on me.

Finally, I want to add a -f option to the script that will read the values that would otherwise be set on the command line. I want to be able to create a single file of repos to migrate so it can be done in a loop.

Actions #12

Updated by Hammel over 1 year ago

  • % Done changed from 30 to 60

The script is working to create orgs (re: groups) and migrate repos using a -f masterfile where each line is a config file for a repo to migrate. This will allow me to create a complete list of my repos to migrate as well as lists of 3rd party repositories to mirror.

However, I need to find the GitLab API that retrieves the repos description so it can be added to the GOGS repo migration.

Note that all migrations are mirrors at this point. That's fine for now.

Update:
Try get single project API to get the description. Note that I need the project ID or name (url-encoded) in this API. I have 10 groups (not all active) on GitLab and I'll need to mirror them all on GOGS which means finding the project ID for all projects in all groups. It also means adding a GITLAB_PROJECT_ID variable to each config file. But hey, I only have to do this once, really.

Mirroring 3rd party repositories will be manual. In some cases, like GTK2, I may have to recreate a repository from source.

Update:
Or maybe not

Actions #13

Updated by Hammel over 1 year ago

  • % Done changed from 60 to 80

Fixed the description retrieval. The script now creates the repos as mirrors with descriptions.

All that's needed now is to create the set of config files for all my repos.

Then create a masterfile for all the 3rd party repos I want to mirror.

Once those are created and tested locally, I can setup Gogs on the colo server and run the script.

Then, finally, I can start creating new repos of all my old software instead of just dumping it as compressed archives in the archives tree.

Actions #14

Updated by Hammel over 1 year ago

Gogs is up and running on the muse site: http://graphics-muse.org:3000/

It's not a secure site, but it's running. I might be able to use existing certs since I'm using an existing domain (but different port).

Actions #15

Updated by Hammel over 1 year ago

  • Severity changed from 03 - Medium to 01 - Critical
Actions #16

Updated by Hammel over 1 year ago

  • % Done changed from 80 to 90

The configs on the muse site have been updated to use the graphics-muse site instead of the local host I used for testing.
All that's left is to run the migration script with the masterfiles and updated configs.

At that point all my repos from gitlab will be mirrored on my gogs site and I can start (outside of this issue) to migrate my old source to the Gogs system and start to mirror projects I think are in danger of disappearing.

Actions #17

Updated by Hammel over 1 year ago

  • Status changed from In Progress to Closed
  • % Done changed from 90 to 100

All non-empty repositories have been migrated as mirrors.
The Gogs interface is now linked from the Muse web site. It will be added to PiboxProject tonight.
Closing issue.

Actions #18

Updated by Hammel 21 days ago

  • Status changed from Closed to In Progress
  • % Done changed from 100 to 50

Gogs was hacked, possibly due to not properly configuring admin access. The hackers did a DoS by filling my disk with projects. I had to remove them manually from the server. I disabled Gogs.

I found I could mirror GitHub projects on GitHub. I've done that with the Raspberry Pi repos.

The next option I found is to fork projects into the public GitLab service under my account. I've done that with Raspberry Pi Linux, Userland and omxplayer. Others will come later.

The next thing I found is that I can run a private docker instance of GitLab in a container. The setup suggests using docker-compose, but a simpler startup uses docker engine. This runs out-of-the-box and I can map host ports to container ports to hide it a bit. This allows me to, for example on the test instance on galileo, access the UI as

http://galileo.gfxmuse.org:1880

or

https://galileo.gfxmuse.org:1443

The container version could also be used in-home, so I could have a public instance on muse and a private one at home that mirrors repos.

I'm experimenting with the private GitLab instance in docker to make sure I use it correctly. To get a shell in the docker container

sudo docker exec -it gitlab /bin/bash

There is a configuration file under /home/gitlab/gitlab.rb where I can setup use of muse's smtp. After making config changes I run

gitlab-ctl reconfigure

to update the running instance. To run rails commands (such as testing email)

gitlab-rails console
Notify.test_email('destination_email@address.com', 'Message Subject', 'Message Body').deliver_now

I can stop the running instance with

sudo docker stop gitlab
sudo docker rm gitlab

and restart it with a script I wrote in ${HOME}/bin called gitlab.sh . To get the initial password

sudo docker exec -it gitlab grep 'Password:' /etc/gitlab/initial_root_password

When the time comes, I can upgrade the whole thing fairly easily.

I can also sign up for a newsletter for security updates.

Changing the admin (and my user, which is also an admin) password is easy and it works fine in the docker image.

Actions #19

Updated by Hammel 11 days ago

Gitlab was nearly perfect - until I found you can't mirror from other repositories without paying for use, and the smallest cost is too prohibitive.

So I'm back to Gogs. There is a Gogs docker image too. And it starts up very easily. The way to prevent attacks again would be to isolate the ports to a single (or set of) IP. I can edit or delete these from iptables is needed.

Gogs is a bit more lightweight in both resource usage and features, but that's okay. It has the mirroring feature and that's really what I need. I may be able to setup the cron jobs for the mirroring within the docker image, too.

It's not clear to me yet if I can run this on https, however. The example only does http, but the startup-configuration may allow setting up https too. I just need to try it to find out.

Actions #20

Updated by Hammel 9 days ago

Once I have gogs running in a docker image, I want to disable access from everyone except me.

I can enable an IP filter in a named chain that would allow me to add other IPs later (if I ever trust anyone to help with PiBox) and then disable the chain if needed.

Actions #21

Updated by Hammel 5 days ago

gogs is re-enabled and all gitlab repos are now mirrored there. The server is not enabled by default and will only be up when I manually bring it up.

Now I need to mirror some of the 3rd party repos that I don't want to lose track of. I already have omxplayer, RPi kernel and userland. The following are some other useful repos I want to mirror.
  1. X.org (mostly the server portions)
  2. GTK2 (which can be found as a branch/tag in the full GTK repo), GDK, and friends
  3. cairo
  4. OpenGL via Mesa
  5. ffmpeg (though I don't see that going away anytime soon)
  6. Any of the 3rd party apps I rely on
    1. Matchbox
    2. mongoose
    3. raspi2fb
    4. lcdshow

There are probably others, but this is a good start.

Actions #22

Updated by Hammel 4 days ago

  • Status changed from In Progress to Closed
  • % Done changed from 50 to 100

These are done but X.org and matchbox both have multiple repositories that could be mirrored.

I'll close this for now. If I need to do more, I'll add new issues.

Actions

Also available in: Atom PDF