[{"content":"","date":null,"permalink":"https://oliverjakobi.com/categories/","section":"Categories","summary":"","title":"Categories"},{"content":"","date":null,"permalink":"https://oliverjakobi.com/categories/linux/","section":"Categories","summary":"","title":"Linux"},{"content":"","date":null,"permalink":"https://oliverjakobi.com/tags/mariadb/","section":"Tags","summary":"","title":"Mariadb"},{"content":"","date":null,"permalink":"https://oliverjakobi.com/posts/","section":"Posts","summary":"","title":"Posts"},{"content":"Some few days ago I repeated one of the more common \u0026lsquo;mistakes\u0026rsquo; I find myself making every once in a while: I setup a vm to test something out, install mariadb, set a root password and then immediately forget it.\nWhat follows is usually me searching for the notes I made only to find out that there are none. \u0026#x1f605;\nSo here\u0026rsquo;s my notes on resetting the root password on mariadb, so that I don\u0026rsquo;t have to search for them and maybe remember them in the future:\n\u0026#x26a0;\u0026#xfe0f; This has last been tested with Mariadb 10.5 Necessary steps # Stop the mariadb service systemctl stop mariadb.service Start mariadb in safe mode without networking: mysqld_safe --skip-grant-tables --skip-networking \u0026amp; Login to the service: mysql Reset the password: SET PASSWORD FOR 'root'@'localhost' = PASSWORD('newrootpassword'); Flush privileges: FLUSH PRIVILEGES Logout from service: quit Stop mariadb: mysqladmin -uroot -p shutdown For later versions this should still work, because from what I recall, mysqld_safe is already a symlink to mariadbd-safe for compatibility reasons. It takes some command line arguments and forwards the rest to mysqld, which also is a symlink to mariadbd. And last but not least, mysqladmin is a symlink to mariadb-admin.\nI believe you could as well call these directly, but I have yet to test this.\nBonus #If you want to make sure that the password actually changed, you can run the following query before and after step 4 :\nSELECT * FROM mysql.global_priv WHERE User = 'root';\n","date":"6 November 2024","permalink":"https://oliverjakobi.com/posts/reset-mariadb-root-password/","section":"Posts","summary":"","title":"reset mariadb root password"},{"content":"","date":null,"permalink":"https://oliverjakobi.com/tags/","section":"Tags","summary":"","title":"Tags"},{"content":"","date":null,"permalink":"https://oliverjakobi.com/categories/devops/","section":"Categories","summary":"","title":"DevOps"},{"content":"Hetzner\u0026rsquo;s object storage is just out of beta and it\u0026rsquo;s stable enough to use for common use cases like backup, static file stoarge and hosting, or using it as OpenTofu S3 Storage Backend.\nUnfortunately, some features are not yet available in the UI but will eventually be implemented.\nOne of these features is bucket versioning, which can only be activated using an S3 client like mc, the minio command line utility.\nThis post covers the setup of mc and enabling bucket versioning.\nInstallation #The installation of mc is relatively easy. On my linux machine, I just grab it with curl and store it on my ~/.local/bin/ directory and make it executable. My .local/bin/ directory already exists and is home to some other programs I use or wrote myself. Choose either a directory within your path, or create a new directory with the --create-dirs flag and add it to your path like this:\ncurl https://dl.min.io/client/mc/release/linux-amd64/mc \\ --create-dirs \\ -o $HOME/.local/bin/mc chmod +x $HOME/.local/bin/mc export PATH=$PATH:$HOME/.local/bin (Make sure to export the $PATH variable in your .bashrc/.zshrc, otherwise you will have to export it every time you want to use mc)\nLastly, make sure you can execute mc by running mc --version. This should show something like\nmc version RELEASE.2024-10-08T09-37-26Z (commit-id=cf128de2cf42e763e7bd30c6df8b749fa94e0c10) Runtime: go1.22.8 linux/amd64 Copyright (c) 2015-2024 MinIO, Inc. License GNU AGPLv3 \u0026lt;https://www.gnu.org/licenses/agpl-3.0.html\u0026gt; If you don\u0026rsquo;t yet have a bucket, follow my post about how to to use Hetzner S3 Object Storage as OpenTofu backend, as it covers the steps to create a bucket and credentials. :)\nIf you have your bucket set up and credentials generated, you can add an `alias`` for your bucket. Make sure you don\u0026rsquo;t accidentally have your credentials end up in your shell history. Refer to the documentation of your shell to find out how to temporarily disable it.\nAdd a provider/project alias to mc #mc saves S3 credentials for you, so you don\u0026rsquo;t have to remember them. This is done with what it calls an \u0026ldquo;alias\u0026rdquo;. Let\u0026rsquo;s add an alias for the bucket of my last post:\nUsually, mc just requires you to add the hostname, access key and secret key, but for hetzner, we also need to tell it the s3 api version and turn path off, so that it uses the bucket name as a subdomain, instead of a directory.\nmc alias set hetzner-test-project https://fsn1.your-objectstorage.com GYJTKTEGSZQH0NMYOPIX iKLz5codZqtq5Pyqtjf6wOtp5izBeg5tAJwzlgFY --api \u0026#34;s3v4\u0026#34; --path off Now that you have your alias setup, you can check it out with mc alias list, and it will show you your configuration along with some defaults, that are already set up:\nmc alias list gcs URL : https://storage.googleapis.com AccessKey : YOUR-ACCESS-KEY-HERE SecretKey : YOUR-SECRET-KEY-HERE API : S3v2 Path : dns Src : /home/oliver/.mc/config.json hetzner-test-project URL : https://fsn1.your-objectstorage.com AccessKey : GYJTKTEGSZQH0NMYOPIX SecretKey : iKLz5codZqtq5Pyqtjf6wOtp5izBeg5tAJwzlgFY API : s3v4 Path : off Src : /home/oliver/.mc/config.json local URL : http://localhost:9000 AccessKey : SecretKey : API : Path : auto Src : /home/oliver/.mc/config.json play URL : https://play.min.io AccessKey : Q3AM3UQ867SPQQA43P2F SecretKey : zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG API : S3v4 Path : auto Src : /home/oliver/.mc/config.json s3 URL : https://s3.amazonaws.com AccessKey : YOUR-ACCESS-KEY-HERE SecretKey : YOUR-SECRET-KEY-HERE API : S3v4 Path : dns Src : /home/oliver/.mc/config.json As you can see, the access key, secret key and other information are stored in json file in your home directory.\nYou can then list the buckets for these credentials with mc ls [alias]:\nmc ls hetzner-test-project [2024-10-28 20:49:19 CET] 0B my-hetzner-bucket-x4lk7x/ Adding the bucket name to this command, will show you its content:\nmc ls hetzner-test-project/my-hetzner-bucket-x4lk7x/ [2024-10-30 07:08:17 CET] 4.1KiB STANDARD my-infra.tfstate Sweet. This works, so we can now activate bucket versioning.\nEnable bucket versioning #Let\u0026rsquo;s first check, whether versioning is already enabled:\nmc version info hetzner-test-project/my-hetzner-bucket-x4lk7x hetzner-test-project/my-hetzner-bucket-x4lk7x is un-versioned It\u0026rsquo;s not, so we can go ahead and enable it:\nmc version enable hetzner-test-project/my-hetzner-bucket-x4lk7x hetzner-test-project/my-hetzner-bucket-x4lk7x versioning is enabled Listing versions with mc #Now let\u0026rsquo;s check, whether versioning works:\nmc ls --versions hetzner-test-project/my-hetzner-bucket-x4lk7x [2024-10-30 07:08:17 CET] 4.1KiB STANDARD null v1 PUT my-infra.tfstate You can see one version of my-infra.tfstate,together with its creation date and size.\nAfter an update to the OpenTofu state, this looks a little different:\nmc ls --versions hetzner-test-project/my-hetzner-bucket-x4lk7x [2024-11-01 11:28:35 CET] 4.8KiB STANDARD .sDyXwoNffIWn-W5XOToETVlMfcaHQF v2 PUT my-infra.tfstate [2024-10-30 07:08:17 CET] 4.1KiB STANDARD null v1 PUT my-infra.tfstate Now you can see both versions, their dates, sizes and even their version ID.\nThat\u0026rsquo;s it! Versioning is now enabled for this bucket.\nConclusion #Enabling versioning on a Hetzner S3 bucket is currently only possible with an S3 client like mc. To install it, refer to the minio client installation documentation for more information on this.\nThere\u0026rsquo;s a lot more you can do with mc, obviously.\nOne word of caution: All versions will count towards your data usage. Keeping versions of many files that change regularly might add up and increase cost for your buckets. Keep that in mind and thanks for reading! \u0026#x1f604;\n","date":"1 November 2024","permalink":"https://oliverjakobi.com/posts/enable-versioning-hetzner-s3-bucket/","section":"Posts","summary":"\u003cp\u003eHetzner\u0026rsquo;s object storage is just out of beta and it\u0026rsquo;s stable enough to use for common use cases like backup, static file stoarge and hosting, or using it as OpenTofu S3 Storage Backend.\u003c/p\u003e\n\u003cp\u003eUnfortunately, some features are not yet available in the UI but will eventually be implemented.\u003c/p\u003e","title":"Enable versioning on Hetzner S3 Object Storage buckets"},{"content":"","date":null,"permalink":"https://oliverjakobi.com/tags/hetzner/","section":"Tags","summary":"","title":"Hetzner"},{"content":"","date":null,"permalink":"https://oliverjakobi.com/tags/s3/","section":"Tags","summary":"","title":"S3"},{"content":"","date":null,"permalink":"https://oliverjakobi.com/categories/hosting/","section":"Categories","summary":"","title":"Hosting"},{"content":"OpenTofu (just like terraform) supports multiple backends for storing your state. For quite a while I kept my state as files on my desktop machine, because running a dedicated database server seemed a bit excessive to me.\nWhen Hetzner announced the beta test phase for their S3 Object Storage offering, I just had to try it out and make my OpenTofu state a bit more resilient and easier to recover. \u0026#x1f62c;\nAdding a bucket to your project #Adding a bucket to one of your projects is done in less than a minute.\nFirst, navigate to the cloud project that you want your bucket to belong to. Then chose Object Storage from the menu on the left.\nNext, click Bucket erstellen (this should be \u0026ldquo;create bucket\u0026rdquo; on the english interface version).\nThis opens a dialog, prompting you to input some information about the new bucket, such as a location, name, and access type.\nDuring the beta phase, only Falkenstein was made available. This should now also be available in Nuremberg and Helsinki (since this post will be released with the official end of the beta test). Your bucket name is required to be unique, contain only lower case characters, digits and hyphens. It must start and end with a character or digit.\nFinally, choose whether you want the bucket to be private or publicly availabe (can be used for static pages, or download storage, etc.)\nYour bucket will then have the format of https://[bucket_name].[location].your-objectstorage.com, just as shown in the dialog.\nFor a bucket name I strongly recommend to use an additional random string as prefix or suffix to your bucket name. This is not a requirement, nor is it a replacement for access permissions, but it will at least help a bit when it comes to folks trying to iterate over dictionaries or common bucket names to find possible data leaks due to wrong permissions on buckets. In the screenshot you can see I appended -x4lk7x to the bucket name my-hetzner-bucket.\nGenerate credentials #In order to access a private bucket, you will need to have credentials. If you just created your bucket, you should now be presented with the list view of all your buckets. Click on the one you have (or want to configure, if there are more than one).\nThis will show you a detail page of the selected bucket. Under \u0026ldquo;S3-Zugangsdaten\u0026rdquo; in the lower left corner, click \u0026ldquo;Zugangsdaten verwalten\u0026rdquo; (manage credentials).\nThis will forward you to the \u0026ldquo;security\u0026rdquo; section of your project, with the \u0026ldquo;S3-Zugangsdaten\u0026rdquo; tab selected.\nClick on \u0026ldquo;Zugangsdaten generieren\u0026rdquo; (generate credentials).\nAfter that, a new dialog prompts you to add a description for the generated credentials. It also informs that the credentials are valid for every bucket in this project, so keep that in mind!\nEnter a description (opentofu-state-credentials for example) and klick \u0026ldquo;Zugangsdaten generieren\u0026rdquo; (generate credentials).\nNow finally, your centials will be displayed, but only once! After that, the secret key will not be displayed any more.\nMake sure to save them first, before you use \u0026rsquo;em \u0026#x1f609;\nConfiguring opentofu to use the s3 backend #To now use your new and shiny bucket, tell opentofu how to use it.\n/my-infra-project/providers.tofu:\nterraform { backend \u0026#34;s3\u0026#34; { bucket = \u0026#34;my-hetzner-bucket-x4lk7x\u0026#34; # Name of your S3 bucket endpoint = \u0026#34;https://fsn1.your-objectstorage.com\u0026#34; # Hetzner\u0026#39;s endpoint key = \u0026#34;my-infra.tfstate\u0026#34; # Name of the tfstate file region = \u0026#34;main\u0026#34; # this is required, but will be skipped! skip_credentials_validation = true # this will skip AWS related validation skip_metadata_api_check = true skip_region_validation = true } required_providers { proxmox = { source = \u0026#34;bpg/proxmox\u0026#34; version = \u0026#34;0.66.3\u0026#34; } [... shortened! ...] } } As you can see in the example above, the region variable is set to a nonsensical value. Since it is required, we need to set it anyway. But it will be disregarded by setting skip_region_validation = true.\nAfter that, there\u0026rsquo;s only one more thing to do: \u0026ldquo;storing\u0026rdquo; the credentials. This is always a hassle, but we\u0026rsquo;ll take care of that now.\nHandling credentials #In general, there are multiple ways you can use the credentials:\nadd them with the access_key and secret_key variables directly into your providers.tofu file (not recommended!) add them to a separate hetzner_s3_credentials.tfvars file and run tofu init -backend-config=hetzner_s3_credentials.tfvars (not recommended!) add the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to your environment add an aws provider and load ~/.aws/credentials TF_VAR_ prefixed environment variables (which opentofu will recognize automatically) don\u0026rsquo;t work in this case, because the backend configuration does not accept variables. Trying to use something like access_key = var.hetzner_s3_access_key will result in an error.\nAdding the credentials to a .tfvars file seems like an easy solution and it sure is. The problem with this approach is, that your credentials will be leaked into ./terraform/terraform.tfstate and plans that you save either on your local machine or when running automated pipelines.\nThis \u0026ldquo;might\u0026rdquo; not seem like a problem, but data ends up in public git repos quicker than you think and we don\u0026rsquo;t really want that, do we?\nAdding an aws provider seems a bit excessive. Also, if you already have it configured, you\u0026rsquo;ll probably be using AWS S3 and have no reason to read this text. \u0026#x1f602;\nSo we\u0026rsquo;re left with environment variables.\nIn s3.env set:\nexport AWS_ACCESS_KEY_ID=GYJTKTEGSZQH0NMYOPIX export AWS_SECRET_ACCESS_KEY=iKLz5codZqtq5Pyqtjf6wOtp5izBeg5tAJwzlgFY Now, the file name here is only an example to show one possible way of doing it. You can just source s3.env and be done with it. However, this still is a little dangerous, if you keep it in your infrastructure repository, even if you remember to configure your .gitignore to ignore the file name.\nYou can also set these variables in your .bashrc, .zshrc, your .profile, or in a separate file that get\u0026rsquo;s sourced by you or a task runner just before planning or applying.\nHowever, this way the credentials will not be stored in the .terraform subdirectory or a plan file. Yay!\nNow you can run tofu init and should see something like this:\nInitializing the backend... Successfully configured the backend \u0026#34;s3\u0026#34;! OpenTofu will automatically use this backend unless the backend configuration changes. Once you run opentofu it will create a statefile and directly push it to the s3 bucket.\nCongrats!\nYou can now head back to the Hetzner Cloud UI and check that everything worked. Navigate to [your project] - Object Storage, select your bucket and click on Dateien (files) tab in the top navigation of your bucket.\nAnd yes, I made sure all data related to this post was deleted once it was written \u0026#x1f609;\nBonus: Enable versioning on your bucket #Having your state saved in object storage is good for easy access with multiple workstations/people. It\u0026rsquo;s even better if you enable versioning on your bucket, so you can still retrieve it, should it be deleted by accident.\n","date":"1 November 2024","permalink":"https://oliverjakobi.com/posts/opentofu-s3-backend-hetzner-storage/","section":"Posts","summary":"","title":"How to use Hetzner S3 Object Storage as OpenTofu backend"},{"content":"","date":null,"permalink":"https://oliverjakobi.com/tags/opentofu/","section":"Tags","summary":"","title":"OpenTofu"},{"content":"","date":null,"permalink":"https://oliverjakobi.com/tags/terraform/","section":"Tags","summary":"","title":"Terraform"},{"content":"Hi! My name is Oliver. \u0026#x1f642; #I am a DevOps Engineer from northern Germany.\nI launched this blog in March 2024 as a pet project.\nIt is mainly focused on documenting my learnings on technical aspecsts of my job and hobby related topics.\nYou will probably find posts about git, ansible, automation in general, lunar photography, sourdough bread or any other thing that sparks my interest.\nCheers!\n","date":null,"permalink":"https://oliverjakobi.com/","section":"","summary":"","title":""},{"content":"This blog post is about me spending three evenings and a night with bad sleep to figure out a misconfiguration I made on my gitlab instance.\nTL;DR #If you set nginx['listen_addresses'] for one of your nginx vhosts in gitlab, make sure to set it for others too, like the registry_nginx['listen_addresses'] setting, applying the ip addresses to listen on for the registry reverse proxy. \u0026#x1f605;\nThe idea #A few days ago I decided to build a CI pipeline in my selfhosted gitlab instance. I had postponed that task for far too long and still built some containers on the cli on one of my docker hosts, where they were supposed to run anyway.\n\u0026ldquo;No bueno\u0026rdquo;, but I guess it\u0026rsquo;s like with so many other things. What you do at work, you sometimes don\u0026rsquo;t do at home. \u0026#x1f937;\nI decided that my container registry should live under it\u0026rsquo;s own subdomain, let\u0026rsquo;s call it registry.example.com, while the gitlab instance lives at git.example.com.\nUsually, this would be a five minute task. It took me 3 evenings and a night with bad sleep until I got it to work.\nConfiguring the gitlab container registry #Luckily, gitlab Documentation is quite extensive and usually easy to understand. Configuring the registry usually only requires you to have a valid TLS certificate at hand (zeroSSL, let\u0026rsquo;s encrypt, or self signed does not matter).\nThen you set registry_external_url 'https://registry.example.com', and registry_nginx['ssl_certificate'] and registry_nginx['ssl_certificate_key'] to point to your TLS cert and key in gitlab.rb.\nNow google \u0026ldquo;how to exit vim\u0026rdquo; for a few minutes, and then run gitlab-ctl reconfigure to apply the configuration changes.\nThis would usually do the trick.\nMy container registry was unavailable #In my case, the container registry was unavailable. When I navigated to https://registry.example.com in my web browser, or called it with curl, I received a redirect to the gitlab login page, but with the registry subdomain.\n😀 oliver@x260  ~  curl -IL -X GET https://registry.example.com HTTP/2 302 server: nginx date: Wed, 09 Oct 2024 05:51:24 GMT content-type: text/html; charset=utf-8 location: https://registry.example.com/users/sign_in \u0026lt;-- why?! [...] removed for readability [...] HTTP/2 200 server: nginx date: Wed, 09 Oct 2024 05:51:24 GMT content-type: text/html; charset=utf-8 content-length: 9391 [...] removed for readability [...] Odd. A 302 redirect to the location https://registry.example.com/users/sign_in. This was clearly wrong. The registry does not provide a login screen.\nDebugging embedded nginx #I\u0026rsquo;m not necessarily friends with nginx, but I used to work with it for a few years, so at least I know my way around.\nGitlab Omnibus installation runs it\u0026rsquo;s services embedded, not as systemd services. ps utility showed that the nginx binary is startet from /opt/gitlab/embedded/sbin/nginx and gets it\u0026rsquo;s configuration prefix passed as -p /var/opt/gitlab/nginx, where it\u0026rsquo;s configuration files reside.\nFirst, I checked the conf directory, which keeps nginx.conf, gitlab-http.conf and gitlab-registry.conf, as well as some additional files for nginx status and gitlab health endpoints. So at least the registry config was there.\nA quick look into nginx.conf showed, that the file was indeed included:\n[...] removed for readability [...] # Enable vts status module. vhost_traffic_status_zone; upstream gitlab-workhorse { server unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket; } include /var/opt/gitlab/nginx/conf/gitlab-http.conf; include /var/opt/gitlab/nginx/conf/gitlab-registry.conf; include /var/opt/gitlab/nginx/conf/nginx-status.conf; However, the gitlab-registry.conf file did not contain any errors. Certificates were configured, proxy configuration was fine.. Nothing that caught my eye.\nThe next step was finding out, which access log showed my browser/curl requests.\nGitlab writes the logs of its services to /var/log/gitlab/[service]/[something].log.\nAs you can already tell from the curl output above, the requests were logged in gitlab_access.log.\nTest all the things! #That was odd. Everything seemed to be right, but my requests were caught in the wrong nginx config file. To double-check I enabled the debug log, by setting error_log /var/log/gitlab/nginx/gitlab_error.log debug;, which spit\u0026rsquo;s out very detailed information about the request and nginx\u0026rsquo;s decisions on where to route it. This also seemed correct, albeit in the wrong virtual host.\nThe following two days I could only spend short time in the evenings to try different things. The short version is:\nI disabled https, no difference I enabled http2, no difference I changed different settings in gitlab.rb, no difference I read and re-read the nginx documentation on host name matching, nothing helped I looked into forgejo and I even thaught about just running a standalone registry I tried debugging the host name matching some more, by changing the host name of the registry Nothing of all of that worked. I went to bed slightly frustrated and didn\u0026rsquo;t sleep very well.\nThe solution #The next morning I remembered that there was one difference in the two configuration files.\ngitlab-http.conf had ips on the listen directives of each server block. gitlab-registry.conf didn\u0026rsquo;t. So this must have been it. Right?\nRight!\nI updated gitlab.rb to set the same ip addresses for registry_nginx['listen_addresses'] that I had set for nginx['listen_addresses] a long time ago to restrict the interfaces, on which nginx should listen, ran gitlab-ctl reconfigure and everything worked! Whoop-whoop! \u0026#x1f389;\nI had completely forgotten that nginx not only tries to match hostnames and uri\u0026rsquo;s. If you have explicitly set ip addresses in one vhost, but omit doing so in the other, nginx will always match to that IP, no matter what.\nWell, I\u0026rsquo;ll hopefully remember now! \u0026#x1f602;\n","date":"9 October 2024","permalink":"https://oliverjakobi.com/posts/nginx-configuration-error-gitlab-container-registry/","section":"Posts","summary":"\u003cp\u003eThis blog post is about me spending three evenings and a night with bad sleep to figure out a misconfiguration I made on my gitlab instance.\u003c/p\u003e\n\u003ch2 id=\"tldr\" class=\"relative group\"\u003eTL;DR \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#tldr\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h2\u003e\u003cp\u003eIf you set \u003ccode\u003enginx['listen_addresses']\u003c/code\u003e for one of your nginx vhosts in gitlab, make sure to set it for others too, like the \u003ccode\u003eregistry_nginx['listen_addresses']\u003c/code\u003e setting, applying the ip addresses to listen on for the registry reverse proxy. \u0026#x1f605;\u003c/p\u003e","title":"a configuration error on nginx made my gitlab container registry unavailable"},{"content":"","date":null,"permalink":"https://oliverjakobi.com/tags/gitlab/","section":"Tags","summary":"","title":"Gitlab"},{"content":"","date":null,"permalink":"https://oliverjakobi.com/tags/maintenance/","section":"Tags","summary":"","title":"Maintenance"},{"content":"","date":null,"permalink":"https://oliverjakobi.com/tags/mdadm/","section":"Tags","summary":"","title":"Mdadm"},{"content":"","date":null,"permalink":"https://oliverjakobi.com/tags/raid1/","section":"Tags","summary":"","title":"Raid1"},{"content":"From time to time, even the best drives go bad and need to be replaced. Here\u0026rsquo;s my notes for doing this without screwing things up.\nRaid1 means we have two drives, for the sake of this note (and because it\u0026rsquo;s mostly true for my servers), these are /dev/sda and /dev/sdb.\nPlease: If you follow this note, make sure to double- or tripple-check your device names and consider additional research before taking action on important infrastructure! This is not a tutorial, nor a \u0026ldquo;definite guide to\u0026hellip;\u0026rdquo; \u0026#x261d;\u0026#xfe0f;\nIn all cases: review configuration #The easiest way for me is always the listing in /proc/mdstat. It shows you, which disk is assigned to which md device and also, which one has gone bad.\ncat /proc/mdstat Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] md1 : active raid1 sdb2[1] sda2[0] 999021888 blocks super 1.2 [2/2] [U_] bitmap: 7/8 pages [28KB], 65536KB chunk md0 : active raid1 sdb1[1] sda1[0] 1046528 blocks super 1.2 [2/2] [U_] unused devices: \u0026lt;none\u0026gt; In this case, if we follow the lines that start with md1 and md0, which are the raid devices in this machine, we can see that sdb2[1] and sda2[0] are part of the raid device. The notation [U_] at the end of the next line shows us the missing/down raid member. These have the save order as they are listed in line above, so sda2 is the defective member in md1 and sda1 is the defective member in md0.\n\u0026#x1f4a1; In [U_], U depicts an \u0026ldquo;Up\u0026rdquo; device and _ depicts a \u0026ldquo;down\u0026rdquo; or \u0026ldquo;missing\u0026rdquo; device. Another easy way would be to lookup devices with lsblk, but that will only work, if the device is still recognized by the os, which they sometimes are not.\nIf /dev/sda has gone bad: #fail and remove the disk from the md device #I always do this. Even if the disk I have to replace is not recognized by the os anymore. There\u0026rsquo;s never been issues with this approach, but I can at least provide anecdotal evidence for it\u0026rsquo;s help in the matter. \u0026#x1f937;\nFail the disk we want to replace:\nmdadm --manage /dev/md0 --fail /dev/sda1 mdadm --manage /dev/md1 --fail /dev/sda2 Now actually remove the disk, you want to replace:\nmdadm --manage /dev/md0 --remove /dev/sda1 mdadm --manage /dev/md1 --remove /dev/sda2 You can then replace the faulty disk.\nOnce the disk is replaced and the server back up and running, continue with the next step:\nCopy partition table from healthy disk to new, empty disk #Be careful with this next step! Take your time, check twice. If you mix up the device names, you end up nuking the partition table on the healthy disk. No bueno!\nCopy partition table from healty disk (here: /dev/sdb) to new disk (here: /dev/sda):\nsfdisk -d /dev/sdb | sfdisk /dev/sda sfdisk -d dumps the partition table to stdout. The pipe hands it back into sfdisk on stdin, which then writes the partition table to the new disk.\nRemove possibly existing superblocks #With this step we make sure that there are no superblocks on the new disk that contain residual data.\nmdadm --zero-superblock /dev/sda1 mdadm --zero-superblock /dev/sda2 Add the new disk to the raid #mdadm -a /dev/md0 /dev/sda1 mdadm -a /dev/md1 /dev/sda2 Your disk should now start resilvering. You can check the status with cat /proc/mdstat.\nInstall grub-config on all disks #You never know, which disk fails, so it\u0026rsquo;s a good idea to have the boot information present on all of them, or booting might fail.\ngrub-install /dev/sda grub-install /dev/sdb Done.\nSome notes on NVMe # NVMe drives are usually named /dev/nvme0n1, /dev/nvme1n1 etc. NVMe partitions are usually named /dev/nvme0n1p1, /dev/nvme0n1p2, where the p marks the partition Your OS might require a reboot to recognize the new partitions on a disk ","date":"6 August 2020","permalink":"https://oliverjakobi.com/posts/replace-drive-raid1-linux/","section":"Posts","summary":"","title":"replace a disk in a raid1 array on linux"},{"content":"Angaben gemäß § 5 TMG #Oliver Jakobi\nLerchenweg 1\n25355 Barmstedt\nKontakt\nTelefon: 04123 806 99 29\nE-Mail: blog@oliverjakobi.com\nVerantwortlich für den Inhalt nach § 55 Abs. 2 RStV #Oliver Jakobi\nLerchenweg 1\n25355 Barmstedt\nHaftung für Inhalte #Als Diensteanbieter sind wir gemäß § 7 Abs.1 TMG für eigene Inhalte auf diesen Seiten nach den allgemeinen Gesetzen verantwortlich. Nach §§ 8 bis 10 TMG sind wir als Diensteanbieter jedoch nicht verpflichtet, übermittelte oder gespeicherte fremde Informationen zu überwachen oder nach Umständen zu forschen, die auf eine rechtswidrige Tätigkeit hinweisen.\nVerpflichtungen zur Entfernung oder Sperrung der Nutzung von Informationen nach den allgemeinen Gesetzen bleiben hiervon unberührt. Eine diesbezügliche Haftung ist jedoch erst ab dem Zeitpunkt der Kenntnis einer konkreten Rechtsverletzung möglich. Bei Bekanntwerden von entsprechenden Rechtsverletzungen werden wir diese Inhalte umgehend entfernen.\nHaftung für Links #Unser Angebot enthält Links zu externen Websites Dritter, auf deren Inhalte wir keinen Einfluss haben. Deshalb können wir für diese fremden Inhalte auch keine Gewähr übernehmen. Für die Inhalte der verlinkten Seiten ist stets der jeweilige Anbieter oder Betreiber der Seiten verantwortlich. Die verlinkten Seiten wurden zum Zeitpunkt der Verlinkung auf mögliche Rechtsverstöße überprüft. Rechtswidrige Inhalte waren zum Zeitpunkt der Verlinkung nicht erkennbar.\nEine permanente inhaltliche Kontrolle der verlinkten Seiten ist jedoch ohne konkrete Anhaltspunkte einer Rechtsverletzung nicht zumutbar. Bei Bekanntwerden von Rechtsverletzungen werden wir derartige Links umgehend entfernen.\nUrheberrecht #Die durch die Seitenbetreiber erstellten Inhalte und Werke auf diesen Seiten unterliegen dem deutschen Urheberrecht. Die Vervielfältigung, Bearbeitung, Verbreitung und jede Art der Verwertung außerhalb der Grenzen des Urheberrechtes bedürfen der schriftlichen Zustimmung des jeweiligen Autors bzw. Erstellers. Downloads und Kopien dieser Seite sind nur für den privaten, nicht kommerziellen Gebrauch gestattet.\nSoweit die Inhalte auf dieser Seite nicht vom Betreiber erstellt wurden, werden die Urheberrechte Dritter beachtet. Insbesondere werden Inhalte Dritter als solche gekennzeichnet. Sollten Sie trotzdem auf eine Urheberrechtsverletzung aufmerksam werden, bitten wir um einen entsprechenden Hinweis. Bei Bekanntwerden von Rechtsverletzungen werden wir derartige Inhalte umgehend entfernen.\nQuelle: https://www.e-recht24.de/impressum-generator.html\n","date":"1 January 0001","permalink":"https://oliverjakobi.com/impressum/","section":"","summary":"","title":"Impressum"}]