articles/git-setup-own-server.md

   1 ---
   2 title: "Git Setup Your Own Server"
   3 date: 2021-06-13T12:07:49+02:00
   4 type: post
   5 author: "danix"
   6 excerpt: "I'll show you how to setup your own git server easily"
   7 featured_image: "uploads/2018/07/gitout.jpg"
   8 tags: ["git", "server", "setup", "howto", "do it yourself", "ssh"]
   9 categories: ["code", "diy", "linux"]
  10 ---
  11
  12 Hello everyone,
  13
  14 I recently decided to move all my code under GIT, I've used it before and I've used also SVN, but I find GIT to be more straightforward in some aspects.
  15
  16 In order to use git I needed a place online where to store my projects, and I thought that github could be a good place, but the fact that you have to pay to keep a project private just didn't sound right in my opinion. Of course github is there to make money (specially now that M$ bought it), but I prefer to have a simpler setup and be able to do things my way as much as possible.
  17
  18 So I started planning what I wanted my git server to have. Here's a list:
  19
  20   * **Security** - I decided to make it work only under ssh, that way only someone who has the key can clone or access the repository. I also added an unprivileged git user that has only access to very few commands, so even if somebody manages to access through ssh he'll find himrself with only very few options available.
  21   * **Notifications** - my server already tells me a lot of what happens, so I wanted my git service to do the same. I implemented a mail service that notifies me every time a new repository is added or everytime there's a push to a repository.
  22   * **Automation** - I wanted to have less steps possible between creation of the project and deployment to production. Now in two steps I can create a repository and clone it to my local computer, and when I'm done I just need to push my modifications and the code is deployed automatically.
  23   * **Visibility** - I haven't yet decided if I want my code to be visible, so I haven't even started thinking about this possibility.
  24
  25 <!--more-->
  26
  27 ## Installation
  28
  29 Installing a git server is quite simple once you know how it works, on my server it was a matter of having a bare repository setup but in order to have the level of security that I wanted there were a few steps involved.
  30
  31 > A bit of a disclaimer here, I use a Slackware64-14.2 on my server and Slackware64-current on my laptop, so all the commands here worked for me but I can't be sure if the procedures that I followed will work on different distros with different setups. If you have any trouble following what I've done let me know in the comments and I'll try to help you.
  32
  33 I've added a new user and group to my server but before doing so I added /usr/bin/git-shell to /etc/shells in order to use it as login shell for my git user.
  34
  35 ```
  36 echo "/usr/bin/git-shell" >> /etc/shells
  37 groupadd git
  38 mkdir /var/git
  39 useradd -d /var/git -g git -M -s /usr/bin/git-shell
  40 ```
  41
  42 now the user is all set and ready to be used. Next step will be to create the .ssh directory and the authorized_keys file to hold the keys for the developers that have to access the git server. Here's how I did it:
  43
  44 ```
  45 mkdir /var/git/.ssh
  46 touch /var/git/.ssh/authorized_keys
  47 chown -R git:git /var/git
  48 chmod 0700 /var/git/.ssh
  49 chmod 0600 /var/git/.ssh/authorized_keys
  50 ```
  51
  52 ok, now the files are in place and the permissions are correct for ssh to work well.
  53
  54 Let's head back to my working computer, I created an ssh keypair for my usual user and copied the public key to the authorized_keys file on the server. I won't go into much detail on how to do so, but just a suggestion, keep it without password, it'll be much faster to work later.
  55
  56 Since I have ssh access to the same server for my normal user I used the ~/.ssh/config file on my computer to set a new host that will ease my access routine for the git user as well as my regular user, that's my config (more or less):
  57
  58 ```
  59 cat ~/.ssh/config
  60 Host regular_ssh
  61         HostName server.tld
  62         User myuser
  63         IdentityFile ~/.ssh/id_rsa
  64
  65 Host git_ssh
  66         HostName server.tld
  67         User git
  68         IdentityFile ~/.ssh/git_rsa
  69 ```
  70
  71 Now when I need to access the server with my regular user I'll just run
  72
  73 `ssh regular_ssh`
  74
  75 and when I need to access as git user I'll run
  76
  77 `ssh git_ssh`
  78
  79 and ssh will take care of all the options and start the connection with the correct credentials for me. Neat!
  80
  81 Now that the access for the git user is setup we have one last thing to do before being able to use it. We'll give him only limited commands to use, That way the git user will be even more limited and much more secure. Inside the documentation shipped with git there's a lot of scripts to get you started with this, so we'll copy them inside a special directory called git-shell-commands, like this:
  82
  83 ```
  84 cp -R /usr/doc/git-2.14.4/contrib/git-shell-commands /var/git
  85 chown -R git:git /var/git
  86
  87 ```
  88
  89 Now we have 2 commands inside the git-shell-commands directory, list and help, the first will show all projects inside the /var/git directory and the other will show a simple help text and a list of all the commands available. Now to give you an example of how easy it is to add commands to the git-shell I will create a simple command that acts as the clear command, it will clean the screen, to do so, from inside the /var/git directory I did:
  90
  91 ```
  92 echo $(which clear) > git-shell-commands/clear
  93 chmod 0755 git-shell-commands/clear
  94
  95 ```
  96
  97 and now I have a "clear" command available for my git user. Another useful command will be "create" to add a repository and a "destroy" to remove it. Let's see them in the next page.
  98
  99 <!--nextpage-->
 100
 101 Let's focus on the two most important scripts for our git server, the **create** and the **delete** script:
 102
 103 ### create
 104
 105 <pre class="wp-block-code language-bash"><code>#! /bin/bash
 106
 107 # usage:        create <PROJECT> - create a git bare repository named PROJECT.git
 108 #               this command will setup the repo and send a mail for confirmation.
 109
 110 GITDIR="/var/git"
 111 MULTIMAIL="/usr/doc/git-2.14.4/contrib/hooks/multimail/git_multimail.py"
 112 GITUSER="git"
 113 GITGRP="git"
 114
 115 function is_bare() {
 116         repodir=$1
 117         if "$(git --git-dir="$repodir" rev-parse --is-bare-repository)" = true
 118         then
 119                 true
 120         else
 121                 false
 122         fi
 123 }
 124
 125 function git_init() {
 126         PROJECT=$1
 127         echo "creating project \"${PROJECT}.git\""
 128         if [ ! -d ${GITDIR}/${PROJECT}.git ]; then
 129                 mkdir ${GITDIR}/${PROJECT}.git
 130         fi
 131         cd ${GITDIR}/${PROJECT}.git
 132         git init --bare
 133         mkdir custom-hooks
 134         ln -s $MULTIMAIL custom-hooks/
 135         touch hooks/post-receive
 136         cat > hooks/post-receive <<EOPR
 137 #!/bin/sh
 138 /usr/bin/pee ${GITDIR}/${PROJECT}.git/custom-hooks/deploy.sh \
 139         ${GITDIR}/${PROJECT}.git/custom-hooks/git_multimail.py
 140
 141 EOPR
 142        cat >> config <<EOT
 143
 144 [multimailhook]
 145         mailer = "sendmail"
 146         refchangeShowGraph = true
 147         mailingList = "receiver@someemail.tld"
 148         commitEmailFormat = "html"
 149         htmlInIntro = true
 150         htmlInFooter = true
 151         from = "sender@someemail.tld"
 152         administrator = "admin@someemail.tld"
 153         quiet = true
 154         logFile = "/var/log/multimail.log"
 155         errorLogFile = "/var/log/multimail_err.log"
 156 EOT
 157         touch custom-hooks/deploy.sh
 158         cat > custom-hooks/deploy.sh <<EODP
 159 #!/bin/bash
 160 # Directory where to deploy files from repository
 161 DPTARGET=""
 162 # Directory containing repository
 163 DPGIT_DIR="${GITDIR}/${PROJECT}.git"
 164 # Branch that is going to be deployed to server
 165 DPBRANCH="master"
 166
 167 while read oldrev newrev ref
 168 do
 169         # if DPTARGET is empty we don't want deploy for this project
 170         if [[ ! "" == \$DPTARGET ]]; then
 171                 # let's check that we are deploying to the correct branch
 172                 if [[ \$ref = refs/heads/\${DPBRANCH} ]]; then
 173                         echo "Ref \$ref received. Deploying \${DPBRANCH} branch to production..."
 174                         git --work-tree=\$DPTARGET --git-dir=\$DPGIT_DIR checkout -f $DPBRANCH
 175                         NOW=\$(date +"%d%m%Y-%H%M")
 176                         git tag release_\$NOW \$DPBRANCH
 177                         echo "   /==============================="
 178                         echo "   | DEPLOYMENT COMPLETED"
 179                         echo "   | Target branch: \$DPTARGET"
 180                         echo "   | Target folder: \$DPGIT_DIR"
 181                         echo "   | Tag name     : release_\$NOW"
 182                         echo "   \=============================="
 183                 else
 184                         echo "Ref \$ref received. Doing nothing: only the \${DPBRANCH} branch may be deployed on this server."
 185                 fi
 186         else
 187                 echo "Target directory not declared. Skipping deploy to server."
 188         fi
 189 done
 190
 191 EODP
 192         chmod 0755 hooks/post-receive custom-hooks/deploy.sh
 193         cd ${GITDIR}/
 194         chown -R ${GITUSER}:${GITGRP} ${GITDIR}/${PROJECT}.git
 195         echo "All done, you can now work on \"${PROJECT}.git\""
 196         exit 0
 197 }
 198
 199 if [ ! -z $1 ]; then
 200         PROJECT=$1
 201 else
 202         read -p 'Project name: ' PROJECT
 203 fi
 204
 205 if [ ! -d ${GITDIR}/${PROJECT}.git ]; then
 206         git_init $PROJECT
 207 else
 208         echo "Project directory ${PROJECT}.git already exists."
 209         if [ $(ls -A ${GITDIR}/${PROJECT}.git) ]; then
 210                 if is_bare ${GITDIR}/${PROJECT}.git
 211                 then
 212                         echo "looks like \"${PROJECT}.git\" is an existing git project directory, choose another name."
 213                         exit 171
 214                 else
 215                         echo "\"${PROJECT}.git\" is not empty, I can't create a Git Project in it. Choose another name."
 216                         exit 172
 217                 fi
 218         else
 219                 echo "\"${PROJECT}.git\" is an empty directory. Do you want to initialize a Git Project here? [y/N]"
 220                 read answer
 221                 case $answer in
 222                         Y|y)
 223                                 git_init $PROJECT
 224                                 ;;
 225                         N|n)
 226                                 echo "Aborting due to user request."
 227                                 exit 173
 228                                 ;;
 229                         *)
 230                                 # we assume no as default answer.
 231                                 echo "you said \"$answer\" which I don't understand, so to me is no. Aborting."
 232                                 exit 177
 233                                 ;;
 234                 esac
 235         fi
 236 fi</code></pre>
 237
 238 This is the create script, as you can see it's a bit complex because it will do a few things for me:
 239
 240   * create the bare repository using the argument provided on the command line or asking for a project name
 241   * check before creating the repo, if another repo with the same name exists or if a directory with files in it exists. That way I'll make sure to only create a repo inside an empty directory.
 242   * add a custom-hooks directory that will hold a link to the multimail.py script as well as my deploy.sh script
 243   * create a post-receive script that uses **pee** from the [moreutils][1] project to run multiple scripts at the same hook.
 244   * ensure sane permissions on the whole project directory.
 245   * always assume no (the safest option) when asking the user about the action to take.
 246
 247 So after saving this script as create inside the git-shell-commands directory, we'll give it executable permissions and we can move to the delete script.
 248
 249 <pre class="wp-block-code language-bash"><code># cat create-bare-repo.sh &gt; /var/git/git-shell-commands/create
 250 # chown git:git -R /var/git/git-shell-commands
 251 # chmod 0755 /var/git/git-shell-commands/create</code></pre>
 252
 253 ### delete
 254
 255 Let's see the delete script:
 256
 257 <pre class="wp-block-code language-bash"><code>#! /bin/bash
 258
 259 # usage:        delete <REPOSITORY> - PERMANENTLY delete a repository if existing.
 260 #               CAREFUL, this action cannot be undone. This command will ask for confirmation.
 261
 262 GITDIR="/var/git"
 263
 264 function is_bare() {
 265         repodir=$1
 266         if "$(git --git-dir="$repodir" rev-parse --is-bare-repository)" = true
 267         then
 268                 true
 269         else
 270                 false
 271         fi
 272 }
 273
 274 if [ ! -z $1 ]; then
 275         PROJECT=$1
 276 else
 277         read -p 'Project to delete: ' PROJECT
 278 fi
 279
 280 if [ -d ${GITDIR}/${PROJECT}.git ]; then
 281         if [[ $(ls -A ${GITDIR}/${PROJECT}.git) ]]; then
 282                 if is_bare ${GITDIR}/${PROJECT}.git
 283                 then
 284                         echo "You are going to delete the git repository \"${PROJECT}.git\" Do you really want to continue? Note, this action cannot be reverted. [y/N]"
 285                         read delAnswer
 286                         case $delAnswer in
 287                                 Y|y)
 288                                         rm -rf ${PROJECT}.git
 289                                         ;;
 290                                 N|n)
 291                                         echo "Aborting due to user request."
 292                                         exit 173
 293                                         ;;
 294                                 *)
 295                                         echo "you said \"$delAnswer\" which I don't understand. Assuming No. Aborting."
 296                                         exit 177
 297                                         ;;
 298                         esac
 299                 else
 300                         echo "\"${PROJECT}.git\" doesn't look like a git repository. Check with your System Administrator."
 301                         exit 177
 302                 fi
 303         else
 304                 echo "\"${PROJECT}.git\" is an empty directory, Skipping. Check with your System Administrator."
 305                 exit 177
 306         fi
 307 fi</code></pre>
 308
 309 This script is much simpler than the previous one, it'll accept the name of the project as argument on the command line or will ask for it and will only delete it if it is a proper git repository, otherwise it will just exit with an error code.
 310
 311 I improved the way those scripts recognise a git repository from simply relying on the fact that there's a HEAD file inside the directory they're checking, which wasn't the best option, to using git itself to check if the directory is a bare repository. Much better!!
 312
 313 Since we are here let's modify the help command to make it show a short description of every available command.
 314
 315 <pre class="wp-block-code language-bash"><code>#!/bin/sh
 316
 317 # usage:        help - Lists all the available commands
 318 #               help <command> - Detailled explanation of how "command" works
 319
 320 if tty -s
 321 then
 322         HELPTEXT="Hi $USER, Run 'help' for help, 'help <command>' for specific help on a command, run 'exit' to exit. Available commands:"
 323 else
 324         HELPTEXT="Hi $USER, Run 'help' for help, 'help <command>' for specific help on a command. Available commands:"
 325 fi
 326
 327 cd "$(dirname "$0")"
 328
 329 if [[ ! -z $1 ]]; then
 330         cmd=$1
 331         if [[ -f $cmd && -x $cmd ]]; then
 332                 awk 'NR>=3&&NR<=4' $cmd | cut -c 3-
 333         else
 334                 echo "command \"$cmd\" doesn't exists"
 335         fi
 336 else
 337         echo $HELPTEXT
 338         for cmd in *
 339         do
 340                 case "$cmd" in
 341                 help) ;;
 342                 *) [ -f "$cmd" ] && [ -x "$cmd" ] && echo "$cmd" ;;
 343                 esac
 344         done
 345 fi</code></pre>
 346
 347 The main thing I added is the support for a command line argument, now I'm able to run it by itself and display the usual output with a list of available commands, or followed by a command name to give a brief explanation like this:
 348
 349 <pre class="wp-block-code language-bash"><code>git> help
 350 Hi git, Run 'help' for help, 'help <command>' for specific help on a command, run 'exit' to exit. Available commands:
 351 clear
 352 create
 353 delete
 354 list
 355 git> help create
 356 usage:  create  - create a git bare repository named PROJECT.git
 357         this command will setup the repo and send a mail for confirmation.
 358 git>
 359 </code></pre>
 360
 361 Pretty nice isn't it?! Now it's much more user friendly, and to show the description I used awk and cut to parse the comment at the top of every script I have in the `git-shell-commands` directory.
 362
 363 In the next page we'll see the usual routine I follow when working with this new setup.
 364
 365 <!--nextpage-->
 366
 367 ## My GIT Routine
 368
 369 Let's say I had a new idea for a WordPress plugin, I can't wait to start writing, so the setup of the GIT environment should be as fast as possible. That's where my setup will come in handy. Let's open the terminal, I'll go inside my testing directory and from there I'll run:
 370
 371 <pre class="wp-block-code language-bash"><code>ssh git_ssh 'create awesomePlugin'
 372 creating project "awesomePlugin.git"
 373 Initialized empty Git repository in /var/git/awesomePlugin.git/
 374 All done, you can now work on "awesomePlugin.git"
 375 </code></pre>
 376
 377 The project is created, now I just need to clone it
 378
 379 <pre class="wp-block-code language-bash"><code>git clone ssh://git_ssh:/var/git/awesomePLugin.git
 380 Cloning into 'awesomePlugin'...
 381 warning: Looks like you cloned an empty repository.
 382 </code></pre>
 383
 384 And that's it, I now have a local and a remote copy of my git repository ready to work with.
 385
 386 Let's say I'm working on this plugin and I get to the point where I feel like I can use it on my own blog, I want the deploy to be as fast as the rest of the process was, so I'll ssh as my normal user and modify the `deploy.sh` script inside the `custom-hooks` directory for this project to add the working directory and the branch I want to use for deploy (you can use master, but it's better to use a different branch only for this pourpose, this way you can keep the production code stable and use master for experimenting until ready to deploy)
 387
 388 Inside the deploy.sh script I'll edit those 2 lines:
 389
 390 <pre class="wp-block-code language-bash"><code># Directory where to deploy files from repository
 391 DPTARGET=""
 392 # Branch that is going to be deployed to server
 393 DPBRANCH="master"
 394 </code></pre>
 395
 396 adding `/var/www/wp-content/plugins/awesomePlugin` as `DPTARGET` and production as `DPBRANCH`.
 397
 398 now on my local system I'll add a new branch and use that before committing my stable code.
 399
 400 <pre class="wp-block-code language-bash"><code>git checkout -b production
 401 Switched to a new branch 'production'
 402
 403 git add .
 404
 405 git commit -m "awesomePlugin is even more awesome"
 406 [production (root-commit) a3885a4] awesomePlugin is even more awesome
 407  1 file changed, 1 insertion(+)
 408  create mode 100644 awesomePlugin.php
 409
 410 git push origin production
 411 Counting objects: 3, done.
 412 Writing objects: 100% (3/3), 240 bytes | 240.00 KiB/s, done.
 413 Total 3 (delta 0), reused 0 (delta 0)
 414 remote: Ref refs/heads/production received. Deploying production branch to production...
 415 remote:    /===============================
 416 remote:    | DEPLOYMENT COMPLETED
 417 remote:    | Target branch: /var/www/htdocs/wp-content/plugins/awesomePlugin
 418 remote:    | Target folder: /var/git/awesomePlugin.git
 419 remote:    | Tag name     : release_12072018-1110
 420 remote:    \==============================
 421 To ssh://git_ssh:/var/git/awesomePlugin.git
 422  * [new branch]      production -> production
 423 </code></pre>
 424
 425 And that's it, now my new plugin is ready to go live as soon as I activate it inside my WordPress admin area.
 426
 427 ## Conclusion
 428
 429 That's it for now, I'll add to this post as soon as I decide whether I want my code to be visible or not, but for now this is my setup and it's working greatly for me so far.
 430
 431 If you made it this far I hope you'll spend a couple more minutes to let me know what you think about this setup, if you use something similar or if you had any problems setting this up, I'll try and help you as much as I can of course.
 432
 433 [1]: https://joeyh.name/code/moreutils/