RestoringCloudFrontendFromBackup - Revision history

Admin at 13:24, 25 July 2024

2024-07-25T13:24:41Z

← Older revision		Revision as of 13:24, 25 July 2024
Line 9:		Line 9:
	* /etc/my.cnf		* /etc/my.cnf
	* /etc/my.cnf.d		* /etc/my.cnf.d

			You will find all of them in '''/backup_mnt/backup/BACKUPS/cloud2/'''

	= Procedure to restore the frontend =		= Procedure to restore the frontend =

Admin at 13:23, 25 July 2024

2024-07-25T13:23:41Z

← Older revision		Revision as of 13:23, 25 July 2024
Line 40:		Line 40:
	</pre>		</pre>

	Once these changes have been pushed in Quattor repo, and before doing the aii-shellfe configure and install on the aii server, there are two things to do to avoid problems (note those 2 steps are done when using quat -ri):		Once these changes have been pushed in Quattor repo, and before doing the aii-shellfe configure and install on the aii server, there are two things to do to avoid problems (note those 2 steps are done when using '''quat -ri'''):
	* revoke the SinDES certificate of the machine on the aii server (if you don't do that, no SinDES ACL will be created since there is already a valid certificate for the machine);		* revoke the SinDES certificate of the machine on the aii server (if you don't do that, no SinDES ACL will be created since there is already a valid certificate for the machine);
	* revoke the Puppet certificate on the Puppet master machine.		* revoke the Puppet certificate on the Puppet master machine.

Admin: /* Procedure to restore the frontend */

2024-07-25T13:22:50Z

Procedure to restore the frontend

← Older revision		Revision as of 13:22, 25 July 2024
Line 23:		Line 23:
	object template cloud2.wn.iihe.ac.be;		object template cloud2.wn.iihe.ac.be;
	include 'machine-types/puppet_node';		include 'machine-types/puppet_node';

			# Mounting backup
			include 'config/nfs/common';
			include 'config/ceph/cephfs.backup';

	# software repositories (should be last)		# software repositories (should be last)
	include PKG_REPOSITORY_CONFIG;		include PKG_REPOSITORY_CONFIG;
Line 35:		Line 40:
	</pre>		</pre>

	Once these changes have been pushed in Quattor repo, and before doing the aii-shellfe configure and install on the aii server, there are two things to do to avoid problems :		Once these changes have been pushed in Quattor repo, and before doing the aii-shellfe configure and install on the aii server, there are two things to do to avoid problems (note those 2 steps are done when using quat -ri):
	* revoke the SinDES certificate of the machine on the aii server (if you don't do that, no SinDES ACL will be created since there is already a valid certificate for the machine);		* revoke the SinDES certificate of the machine on the aii server (if you don't do that, no SinDES ACL will be created since there is already a valid certificate for the machine);
	* revoke the Puppet certificate on the Puppet master machine.		* revoke the Puppet certificate on the Puppet master machine.
Line 44:		Line 49:
	<pre>		<pre>
	object template cloud2.wn.iihe.ac.be;		object template cloud2.wn.iihe.ac.be;

	include 'machine-types/puppet_node';		include 'machine-types/puppet_node';

	variable ONE_RELEASE = '6.6';		variable ONE_RELEASE = '6.6';
	include 'features/one_frontend/light_config';		include 'features/one_frontend/light_config';
	include 'features/one_frontend/one6.X/sunstone_apache_ssl';		include 'features/one_frontend/one6.X/sunstone_apache_ssl';

			# Mounting backup
			include 'config/nfs/common';
			include 'config/ceph/cephfs.backup';

			# Making backup of everything that is needed
			include 'components/cron/config';
			'/software/components/cron/entries' = push(
			dict(
			'name', 'backup_config_db_one',
			'user', 'root',
			'frequency', '30 /2 * *',
			'log', dict('disabled', false),
			'command', '/usr/bin/rsync -avR /var/lib/one /var/lib/mysql /etc/one /etc/my.cnf /etc/my.cnf.d /backup_mnt/backup/BACKUPS/cloud2/.'
			)
			);

	# software repositories (should be last)		# software repositories (should be last)
	include PKG_REPOSITORY_CONFIG;		include PKG_REPOSITORY_CONFIG;

Admin: /* Procedure to restore the frontend */

2024-07-25T13:18:07Z

Procedure to restore the frontend

← Older revision		Revision as of 13:18, 25 July 2024
Line 41:		Line 41:
	When the Quattor installation is finished, you have to mount the '''backup''' share on the VM, and then you restore the files and directories with the command of your taste (in my case, I just used '''cp -ap'''). Of course, it is really important to preserve the permissions and ownerships of the source files.		When the Quattor installation is finished, you have to mount the '''backup''' share on the VM, and then you restore the files and directories with the command of your taste (in my case, I just used '''cp -ap'''). Of course, it is really important to preserve the permissions and ownerships of the source files.

	With the data and configuration files being restored, it is now time to switch the VM as a good old OpenNebula frontend. Revert the profile to something like this:		With the data and configuration files being restored, it is now time to switch the VM back as a good old OpenNebula frontend. Revert the profile to something like this:
	<pre>		<pre>
	object template cloud2.wn.iihe.ac.be;		object template cloud2.wn.iihe.ac.be;

Admin: /* Procedure to restore the backup */

2024-07-25T13:11:19Z

Procedure to restore the backup

@@ Line 10: / Line 10: @@
 * /etc/my.cnf.d
-= Procedure to restore the backup =
+= Procedure to restore the frontend =
 On the standalone hypervizor, create a new VM with same hardware characteristics as the previous (mac address, disk size, memory, cpu,...). The easiest way is to copy-paste the xml of the previous. Be careful that libvirt will complain about the fact that you reuse the mac address. The solution is simple: remove the nic of the previous VM to free the mac address. Here are some commands that might be useful for this step:
 <pre>
@@ Line 38: / Line 38: @@
 * revoke the SinDES certificate of the machine on the aii server (if you don't do that, no SinDES ACL will be created since there is already a valid certificate for the machine);
 * revoke the Puppet certificate on the Puppet master machine.

Admin: /* Procedure to restore the backup */

2024-07-25T13:00:40Z

Procedure to restore the backup

← Older revision		Revision as of 13:00, 25 July 2024
Line 17:		Line 17:
	virsh create <xml_description_of_vm>		virsh create <xml_description_of_vm>
	</pre>		</pre>
	Of course, you can also use the GUI '''virt-manager''' for most tasks. Be especially careful with the drivers (it should be '''virtio''' for the NIC and the drive). Also double-check that the drive ~~has~~ the ~~same size and~~ the same ~~amount of memory~~ as the previous VM. And of course, don't reuse the disk of the previous VM, you have to create a new one (it's easy to do from the '''virt-manager''' GUI.		Of course, you can also use the GUI '''virt-manager''' for most tasks. Be especially careful with the drivers (it should be '''virtio''' for the NIC and the drive). Also double-check that the drive and the memory have the same sizes as on the previous VM. And of course, don't reuse the disk of the previous VM, you have to create a new one (it's easy to do from the '''virt-manager''' GUI).

	Once the VM is running, you'll have to reinstall the frontend on it with Quattor and Puppet. However, the VM must initially be reinstalled with machine-type 'puppet_node' and Puppet app set to 'servers' and role to 'none'. Why? Because if you directly reinstall the VM as frontend, the initization scripts that come with the ONE packages and also with Puppet will generate some new settings that might be tricky to overwrite with the backup. So to say, in the beginning of the restore process, the machine must be a vanilla one. Here is the vanilla profile used to reinstall the VM:		Once the VM is running, you'll have to reinstall the frontend on it with Quattor and Puppet. However, the VM must initially be reinstalled with machine-type 'puppet_node' and Puppet app set to 'servers' and role to 'none'. Why? Because if you directly reinstall the VM as frontend, the initization scripts that come with the ONE packages and also with Puppet will generate some new settings that might be tricky to overwrite with the backup. So to say, in the beginning of the restore process, the machine must be a vanilla one. Here is the vanilla profile used to reinstall the VM:

Admin: /* Procedure to restore the backup */

2024-07-25T12:58:57Z

Procedure to restore the backup

← Older revision		Revision as of 12:58, 25 July 2024
Line 17:		Line 17:
	virsh create <xml_description_of_vm>		virsh create <xml_description_of_vm>
	</pre>		</pre>
	Of course, you can also use the GUI '''virt-manager''' for most tasks. Be especially careful with the drivers (it should be '''virtio''' for the NIC and the drive. Also double-check that the drive has the same size and the same amount of memory as the previous VM.		Of course, you can also use the GUI '''virt-manager''' for most tasks. Be especially careful with the drivers (it should be '''virtio''' for the NIC and the drive). Also double-check that the drive has the same size and the same amount of memory as the previous VM. And of course, don't reuse the disk of the previous VM, you have to create a new one (it's easy to do from the '''virt-manager''' GUI.

	Once the VM is running, you'll have to reinstall the frontend on it with Quattor and Puppet. However, the VM must initially be reinstalled with machine-type 'puppet_node' and Puppet app set to 'servers' and role to 'none'. Why? Because if you directly reinstall the VM as frontend, the ~~initizations~~ scripts that come with the ONE packages and also with Puppet will generate some new settings that might be tricky to overwrite with the backup. So to say, in the beginning of the restore process, the machine must be a vanilla one. Here is the vanilla profile used to reinstall the VM:		Once the VM is running, you'll have to reinstall the frontend on it with Quattor and Puppet. However, the VM must initially be reinstalled with machine-type 'puppet_node' and Puppet app set to 'servers' and role to 'none'. Why? Because if you directly reinstall the VM as frontend, the initization scripts that come with the ONE packages and also with Puppet will generate some new settings that might be tricky to overwrite with the backup. So to say, in the beginning of the restore process, the machine must be a vanilla one. Here is the vanilla profile used to reinstall the VM:
	<pre>		<pre>
	object template cloud2.wn.iihe.ac.be;		object template cloud2.wn.iihe.ac.be;
Line 34:		Line 34:
	'cloud', 'cloud2',		'cloud', 'cloud2',
	</pre>		</pre>

			Once these changes have been pushed in Quattor repo, and before doing the aii-shellfe configure and install on the aii server, there are two things to do to avoid problems :
			* revoke the SinDES certificate of the machine on the aii server (if you don't do that, no SinDES ACL will be created since there is already a valid certificate for the machine);
			* revoke the Puppet certificate on the Puppet master machine.

Admin: /* Procedure to restore the backup */

2024-07-25T12:49:57Z

Procedure to restore the backup

@@ Line 17: / Line 17: @@
 virsh create <xml_description_of_vm>
 </pre>
-Of course, you can also use the GUI '''virt-manager''' for most tasks.
+Of course, you can also use the GUI '''virt-manager''' for most tasks. Be especially careful with the drivers (it should be '''virtio''' for the NIC and the drive. Also double-check that the drive has the same size and the same amount of memory as the previous VM.
-Once the VM is running, you'll have to reinstall the frontend on it with Quattor and Puppet. However, the VM must initially be reinstalled with machine-type 'puppet_node' and Puppet app set to 'servers' and role to 'none'. Why? Because if you directly reinstall the VM as frontend, the initizations scripts that come with the ONE packages and also with Puppet will generate some new settings that might be tricky to overwrite with the backup. So to say, in the beginning of the restore process, the machine must be a vanilla one.
+Once the VM is running, you'll have to reinstall the frontend on it with Quattor and Puppet. However, the VM must initially be reinstalled with machine-type 'puppet_node' and Puppet app set to 'servers' and role to 'none'. Why? Because if you directly reinstall the VM as frontend, the initizations scripts that come with the ONE packages and also with Puppet will generate some new settings that might be tricky to overwrite with the backup. So to say, in the beginning of the restore process, the machine must be a vanilla one. Here is the vanilla profile used to reinstall the VM:

Admin: Created page with "= Context = In July 2024, we have lost our OpenNebula frontend VM (cloud2) after an attempt to reboot it. It was hosted on domz02, a basic QEMU/KVM standalone hypervizor managed with libvirt. The problem seemed to be that the system was not able to find the partition table in the qcow2 image. As there was no backup of the OpenNebula data and config files, the only solution to recover these things was to attach the image to a new VM and to recreate the partition table in..."

2024-07-25T12:43:39Z

Created page with "= Context = In July 2024, we have lost our OpenNebula frontend VM (cloud2) after an attempt to reboot it. It was hosted on domz02, a basic QEMU/KVM standalone hypervizor managed with libvirt. The problem seemed to be that the system was not able to find the partition table in the qcow2 image. As there was no backup of the OpenNebula data and config files, the only solution to recover these things was to attach the image to a new VM and to recreate the partition table in..."

New page

= Context =
In July 2024, we have lost our OpenNebula frontend VM (cloud2) after an attempt to reboot it. It was hosted on domz02, a basic QEMU/KVM standalone hypervizor managed with libvirt. The problem seemed to be that the system was not able to find the partition table in the qcow2 image. As there was no backup of the OpenNebula data and config files, the only solution to recover these things was to attach the image to a new VM and to recreate the partition table in the mounted image using the tool gpart. It eventually worked, but things would have far more easier if we had a simple backup of the directories containing the important ONE data and configuration files of our cloud system. And above all, the procedure we followed to restore the machine was sketched in a situation of emergency with no warranty that it would succeed.

= Data and configuration items to backup =
Here is a list of the important files/directories to backup:
* /var/lib/one
* /var/lib/mysql
* /etc/one
* /etc/my.cnf
* /etc/my.cnf.d

= Procedure to restore the backup =
On the standalone hypervizor, create a new VM with same hardware characteristics as the previous (mac address, disk size, memory, cpu,...). The easiest way is to copy-paste the xml of the previous. Be careful that libvirt will complain about the fact that you reuse the mac address. The solution is simple: remove the nic of the previous VM to free the mac address. Here are some commands that might be useful for this step:
<pre>
virsh list
virsh edit <machine_name>
virsh create <xml_description_of_vm>
</pre>
Of course, you can also use the GUI '''virt-manager''' for most tasks.

Once the VM is running, you'll have to reinstall the frontend on it with Quattor and Puppet. However, the VM must initially be reinstalled with machine-type 'puppet_node' and Puppet app set to 'servers' and role to 'none'. Why? Because if you directly reinstall the VM as frontend, the initizations scripts that come with the ONE packages and also with Puppet will generate some new settings that might be tricky to overwrite with the backup. So to say, in the beginning of the restore process, the machine must be a vanilla one.