{"id":381,"date":"2020-11-21T18:23:20","date_gmt":"2020-11-21T18:23:20","guid":{"rendered":"http:\/\/gexperts.com\/wp\/?p=381"},"modified":"2020-11-21T18:23:20","modified_gmt":"2020-11-21T18:23:20","slug":"openshift-home-lab-and-block-storage","status":"publish","type":"post","link":"https:\/\/gexperts.com\/wp\/openshift-home-lab-and-block-storage\/","title":{"rendered":"OpenShift Home Lab and Block Storage"},"content":{"rendered":"

I have a single server (Ryzen 3900x with 128 GB of RAM) homelab environment that I use to run OpenShift (plus it doubles as my gaming PC). The host is running Fedora 32 at the time of this writing and I run OpenShift on libvirt using a playbook<\/a> created by Luis Javier Arizmendi Alonso that sets everything up including NFS storage. The NFS server runs on the host machine and the OpenShift nodes running in VMs access the NFS server via the host IP to provision PVs. Luis’s playbook sets up a dynamic NFS provisioner in OpenShift and it all works wonderfully.<\/p>\n

However there are times when you do need block storage, while NFS is capable of handling some loads that would traditionally require block, small databases for example, I was having issues with some other more intensive workloads like Kafka. Fortunately I had a spare 500 GB SSD lying around from my retired gaming computer and I figured I could drop that into my homelab server and use as block storage. Hence began my journey of learning way more about iscsi then I ever wanted to know as a developer…<\/p>\n

Here are the steps I used to get static block storage going, I’m definitely interested if there are better ways to do it particularly if someone has dynamic block storage going in libvirt then drop me a line! Note these instructions were written for Fedora 32 which is what my host is using. <\/p>\n

The first step is we need to partition the SSD using LVM into chunks that we can eventually serve up as PVs in OpenShift. This process is pretty straightforward, first we need to create a physical volume and a volume group called ‘iscsi’. Note my SSD is on ‘\/dev\/sda’, your mileage will vary so replace the ‘\/dev\/sdX’ below with whatever device you are using. Be careful not to overwrite something that is in use.<\/p>\n

\r\npvcreate \/dev\/sdX\r\nvgcreate iscsi \/dev\/sdX\r\n<\/pre>\n
Next we create a logical volume, I’ve opted to create a thin pool which means that storage doesn’t get allocated until it’s actually used. This allows you to over-provision storage if you need to though obviously some care is required. To create the thin pool run the following:<\/p>\n
\r\nlvcreate -l 100%FREE -T iscsi\/thin_pool\r\n<\/pre>\n
One we have our pool created we then need to create the actual volumes that will be available as PVs. I’ve chosen to create a mix of PV sizes as per below, feel free to vary depending on your use case. Having said that note the naming convention I am using which will flow up into our iscsi and PV configuration, I highly recommend you use a similar convention for consistency.<\/p>\n
\r\nlvcreate -V 100G -T iscsi\/thin_pool -n block0_100\r\nlvcreate -V 100G -T iscsi\/thin_pool -n block1_100\r\nlvcreate -V 50G -T iscsi\/thin_pool -n block2_50\r\nlvcreate -V 50G -T iscsi\/thin_pool -n block3_50\r\nlvcreate -V 10G -T iscsi\/thin_pool -n block4_10\r\nlvcreate -V 10G -T iscsi\/thin_pool -n block5_10\r\nlvcreate -V 10G -T iscsi\/thin_pool -n block6_10\r\nlvcreate -V 10G -T iscsi\/thin_pool -n block7_10\r\nlvcreate -V 10G -T iscsi\/thin_pool -n block8_10\r\n<\/pre>\n
Note if you make a mistake and want to remove a volume, you can do so by running the following command:<\/p>\n
\r\nlvremove iscsi\/block5_50\r\n<\/pre>\n
Next we need to install some iscsi packages onto the host in order to configure and run the iscsi daemon on the host. <\/p>\n
\r\ndnf install iscsi-initiator-utils targetcli\r\n<\/pre>\n
I’ve opted to use targetcli to configure iscsi rather then hand bombing a bunch of files, it provides a nice cli interface over the process which for me, being an iscsi newbie, greatly appreciated. When you run targetcli it wil drop you into a prompt as follows:<\/p>\n
\r\n[gnunn@lab-server ~]$ sudo targetcli\r\n[sudo] password for gnunn: \r\ntargetcli shell version 2.1.53\r\nCopyright 2011-2013 by Datera, Inc and others.\r\nFor help on commands, type 'help'.\r\n\r\n\/> \r\n\r\n<\/pre>\n
The prompt basically follows standard linux file system conventions and you can use commands like ‘cd’ and ‘ls’ to navigate it. The first thing we are going to do is create our block devices which map to our LVM PVs. In the targetcli prompt this is done with the following commands, note the naming convention being used which ties these devices to our PVs:<\/p>\n
\r\ncd backstores\/block\r\ncreate dev=\/dev\/mapper\/iscsi-block0_100 name=disk0-100\r\ncreate dev=\/dev\/mapper\/iscsi-block1_100 name=disk1-100\r\ncreate dev=\/dev\/mapper\/iscsi-block2_50 name=disk2-50\r\ncreate dev=\/dev\/mapper\/iscsi-block3_50 name=disk3-50\r\ncreate dev=\/dev\/mapper\/iscsi-block4_10 name=disk4-10\r\ncreate dev=\/dev\/mapper\/iscsi-block5_10 name=disk5-10\r\ncreate dev=\/dev\/mapper\/iscsi-block6_10 name=disk6-10\r\ncreate dev=\/dev\/mapper\/iscsi-block7_10 name=disk7-10\r\ncreate dev=\/dev\/mapper\/iscsi-block8_10 name=disk8-10\r\n<\/pre>\n
Next we create the initiator in iscsi, note that my host name is lab-server so I used that in the name below, feel free to modify as you prefer. I’ll admit I’m still a little fuzzy on iscsi naming conventions so suggestions welcome from those of you with more experience.<\/p>\n
\r\ncd \/iscsi\r\ncreate iqn.2003-01.org.linux-iscsi.lab-server:openshift\r\n<\/pre>\n
Next we create the luns, the luns map to our block devices and represent the storage that will be available:<\/p>\n
\r\ncd \/iscsi\/iqn.2003-01.org.linux-iscsi.lab-server:openshift\/tpg1\/luns\r\ncreate storage_object=\/backstores\/block\/disk0-100\r\ncreate storage_object=\/backstores\/block\/disk1-100\r\ncreate storage_object=\/backstores\/block\/disk2-50\r\ncreate storage_object=\/backstores\/block\/disk3-50\r\ncreate storage_object=\/backstores\/block\/disk4-10\r\ncreate storage_object=\/backstores\/block\/disk5-10\r\ncreate storage_object=\/backstores\/block\/disk6-10\r\ncreate storage_object=\/backstores\/block\/disk7-10\r\ncreate storage_object=\/backstores\/block\/disk8-10\r\n<\/pre>\n
Next we create the acls which control access to the luns. Note in my case my lab server is running on a private network behind a firewall so I have not bothered with any sort of authentication. If this is not the case for you then I would definitely recommend spending some time looking into adding this.<\/p>\n
\r\ncd \/iscsi\/iqn.2003-01.org.linux-iscsi.lab-server:openshift\/tpg1\/acls\r\ncreate iqn.2003-01.org.linux-iscsi.lab-server:client\r\ncreate iqn.2003-01.org.linux-iscsi.lab-server:openshift-client\r\n<\/pre>\n
Note I’ve created two acls, one as a generic client and one specific for my openshift cluster.<\/p>\n
Finally the last step is the portal, a default portal will be created that binds to all ports on 0.0.0.0. My preference is to remove it and bind it to a specific IP address on the host. My host as two ethernet ports so here I am binding it to the 2.5 gigabit port which has a static IP address, your IP address will obviously vary.<\/p>\n
\r\ncd \/iscsi\/iqn.2003-01.org.linux-iscsi.lab-server:openshift\/tpg1\/portal\r\ndelete 0.0.0.0 ip_port=3260\r\ncreate 192.168.1.83\r\n<\/pre>\n
Once you have done all this, you should have a result that looks similar to the following when you run ‘ls \/’ in targetcli:<\/p>\n
\r\no- \/ ......................................................................................................................... [...]\r\n  o- backstores .............................................................................................................. [...]\r\n  | o- block .................................................................................................. [Storage Objects: 9]\r\n  | | o- disk0-100 .................................................. [\/dev\/mapper\/iscsi-block0_100 (100.0GiB) write-thru activated]\r\n  | | | o- alua ................................................................................................... [ALUA Groups: 1]\r\n  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active\/optimized]\r\n  | | o- disk1-100 .................................................. [\/dev\/mapper\/iscsi-block1_100 (100.0GiB) write-thru activated]\r\n  | | | o- alua ................................................................................................... [ALUA Groups: 1]\r\n  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active\/optimized]\r\n  | | o- disk2-50 ..................................................... [\/dev\/mapper\/iscsi-block2_50 (50.0GiB) write-thru activated]\r\n  | | | o- alua ................................................................................................... [ALUA Groups: 1]\r\n  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active\/optimized]\r\n  | | o- disk3-50 ..................................................... [\/dev\/mapper\/iscsi-block3_50 (50.0GiB) write-thru activated]\r\n  | | | o- alua ................................................................................................... [ALUA Groups: 1]\r\n  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active\/optimized]\r\n  | | o- disk4-10 ..................................................... [\/dev\/mapper\/iscsi-block4_10 (10.0GiB) write-thru activated]\r\n  | | | o- alua ................................................................................................... [ALUA Groups: 1]\r\n  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active\/optimized]\r\n  | | o- disk5-10 ..................................................... [\/dev\/mapper\/iscsi-block5_10 (10.0GiB) write-thru activated]\r\n  | | | o- alua ................................................................................................... [ALUA Groups: 1]\r\n  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active\/optimized]\r\n  | | o- disk6-10 ..................................................... [\/dev\/mapper\/iscsi-block6_10 (10.0GiB) write-thru activated]\r\n  | | | o- alua ................................................................................................... [ALUA Groups: 1]\r\n  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active\/optimized]\r\n  | | o- disk7-10 ..................................................... [\/dev\/mapper\/iscsi-block7_10 (10.0GiB) write-thru activated]\r\n  | | | o- alua ................................................................................................... [ALUA Groups: 1]\r\n  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active\/optimized]\r\n  | | o- disk8-10 ..................................................... [\/dev\/mapper\/iscsi-block8_10 (10.0GiB) write-thru activated]\r\n  | |   o- alua ................................................................................................... [ALUA Groups: 1]\r\n  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active\/optimized]\r\n  | o- fileio ................................................................................................. [Storage Objects: 0]\r\n  | o- pscsi .................................................................................................. [Storage Objects: 0]\r\n  | o- ramdisk ................................................................................................ [Storage Objects: 0]\r\n  o- iscsi ............................................................................................................ [Targets: 1]\r\n  | o- iqn.2003-01.org.linux-iscsi.lab-server:openshift .................................................................. [TPGs: 1]\r\n  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]\r\n  |     o- acls .......................................................................................................... [ACLs: 2]\r\n  |     | o- iqn.2003-01.org.linux-iscsi.lab-server:client ........................................................ [Mapped LUNs: 9]\r\n  |     | | o- mapped_lun0 ............................................................................. [lun0 block\/disk0-100 (rw)]\r\n  |     | | o- mapped_lun1 ............................................................................. [lun1 block\/disk1-100 (rw)]\r\n  |     | | o- mapped_lun2 .............................................................................. [lun2 block\/disk2-50 (rw)]\r\n  |     | | o- mapped_lun3 .............................................................................. [lun3 block\/disk3-50 (rw)]\r\n  |     | | o- mapped_lun4 .............................................................................. [lun4 block\/disk4-10 (rw)]\r\n  |     | | o- mapped_lun5 .............................................................................. [lun5 block\/disk5-10 (rw)]\r\n  |     | | o- mapped_lun6 .............................................................................. [lun6 block\/disk6-10 (rw)]\r\n  |     | | o- mapped_lun7 .............................................................................. [lun7 block\/disk7-10 (rw)]\r\n  |     | | o- mapped_lun8 .............................................................................. [lun8 block\/disk8-10 (rw)]\r\n  |     | o- iqn.2003-01.org.linux-iscsi.lab-server:openshift-client .............................................. [Mapped LUNs: 9]\r\n  |     |   o- mapped_lun0 ............................................................................. [lun0 block\/disk0-100 (rw)]\r\n  |     |   o- mapped_lun1 ............................................................................. [lun1 block\/disk1-100 (rw)]\r\n  |     |   o- mapped_lun2 .............................................................................. [lun2 block\/disk2-50 (rw)]\r\n  |     |   o- mapped_lun3 .............................................................................. [lun3 block\/disk3-50 (rw)]\r\n  |     |   o- mapped_lun4 .............................................................................. [lun4 block\/disk4-10 (rw)]\r\n  |     |   o- mapped_lun5 .............................................................................. [lun5 block\/disk5-10 (rw)]\r\n  |     |   o- mapped_lun6 .............................................................................. [lun6 block\/disk6-10 (rw)]\r\n  |     |   o- mapped_lun7 .............................................................................. [lun7 block\/disk7-10 (rw)]\r\n  |     |   o- mapped_lun8 .............................................................................. [lun8 block\/disk8-10 (rw)]\r\n  |     o- luns .......................................................................................................... [LUNs: 9]\r\n  |     | o- lun0 .............................................. [block\/disk0-100 (\/dev\/mapper\/iscsi-block0_100) (default_tg_pt_gp)]\r\n  |     | o- lun1 .............................................. [block\/disk1-100 (\/dev\/mapper\/iscsi-block1_100) (default_tg_pt_gp)]\r\n  |     | o- lun2 ................................................ [block\/disk2-50 (\/dev\/mapper\/iscsi-block2_50) (default_tg_pt_gp)]\r\n  |     | o- lun3 ................................................ [block\/disk3-50 (\/dev\/mapper\/iscsi-block3_50) (default_tg_pt_gp)]\r\n  |     | o- lun4 ................................................ [block\/disk4-10 (\/dev\/mapper\/iscsi-block4_10) (default_tg_pt_gp)]\r\n  |     | o- lun5 ................................................ [block\/disk5-10 (\/dev\/mapper\/iscsi-block5_10) (default_tg_pt_gp)]\r\n  |     | o- lun6 ................................................ [block\/disk6-10 (\/dev\/mapper\/iscsi-block6_10) (default_tg_pt_gp)]\r\n  |     | o- lun7 ................................................ [block\/disk7-10 (\/dev\/mapper\/iscsi-block7_10) (default_tg_pt_gp)]\r\n  |     | o- lun8 ................................................ [block\/disk8-10 (\/dev\/mapper\/iscsi-block8_10) (default_tg_pt_gp)]\r\n  |     o- portals .................................................................................................... [Portals: 1]\r\n  |       o- 192.168.1.83:3260 ................................................................................................ [OK]\r\n  o- loopback ......................................................................................................... [Targets: 0]\r\n  o- vhost ............................................................................................................ [Targets: 0]\r\n<\/pre>\n
At this point you can exit targetcli by typing ‘exit’ in the prompt. Next at this point we need to expose the iscsi port in firewalld and enable the services:<\/p>\n
\r\nfirewall-cmd --add-service=iscsi-target --permanent\r\nfirewall-cmd --reload\r\nsystemctl enable iscsid\r\nsystemctl start iscsid\r\nsystemctl enable target\r\nsystemctl start target\r\n<\/pre>\n
Note that the target service ensures the configuration you created in targetcli is restored whenever the host is restarted. If you do not enable and start the target the next time the computer starts you will notice an empty configuration.<\/p>\n
Now that the host is created we can go ahead and create the static PVs for OpenShift as well as the non-provisioning storage class. You can view the PVs I’m using in git here<\/a>, I won’t paste them in the blog since it’s a long file. We wrap these PVs in a non-provisioning storage class so we can request them easily on demand from our applications.<\/p>\n
\r\nkind: StorageClass\r\napiVersion: storage.k8s.io\/v1\r\nmetadata:\r\n  name: iscsi\r\nprovisioner: no-provisioning\r\nparameters:\r\n<\/pre>\n
To test out the PVs, here is an example PVS:<\/p>\n
\r\napiVersion: \"v1\"\r\nkind: \"PersistentVolumeClaim\"\r\nmetadata:\r\n  name: \"block\"\r\nspec:\r\n  accessModes:\r\n    - \"ReadWriteOnce\"\r\n  resources:\r\n    requests:\r\n      storage: \"100Gi\"\r\n  storageClassName: \"iscsi\"\r\n<\/pre>\n
And that’s it, now you have access to block storage in your homelab environment. I’ve used this quite a bit with kafka and it works great, I’m looking into doing some benchmarking of this versus AWS EBS to see how the performance compares and will follow up on this in another blog.<\/p>\n","protected":false},"excerpt":{"rendered":"
I have a single server (Ryzen 3900x with 128 GB of RAM) homelab environment that I use to run OpenShift (plus it doubles as my gaming PC). The host is running Fedora 32 at the time of this writing and … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[12],"tags":[],"class_list":["post-381","post","type-post","status-publish","format-standard","hentry","category-openshift"],"_links":{"self":[{"href":"https:\/\/gexperts.com\/wp\/wp-json\/wp\/v2\/posts\/381","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/gexperts.com\/wp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/gexperts.com\/wp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/gexperts.com\/wp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/gexperts.com\/wp\/wp-json\/wp\/v2\/comments?post=381"}],"version-history":[{"count":8,"href":"https:\/\/gexperts.com\/wp\/wp-json\/wp\/v2\/posts\/381\/revisions"}],"predecessor-version":[{"id":389,"href":"https:\/\/gexperts.com\/wp\/wp-json\/wp\/v2\/posts\/381\/revisions\/389"}],"wp:attachment":[{"href":"https:\/\/gexperts.com\/wp\/wp-json\/wp\/v2\/media?parent=381"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/gexperts.com\/wp\/wp-json\/wp\/v2\/categories?post=381"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/gexperts.com\/wp\/wp-json\/wp\/v2\/tags?post=381"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}