.. _Appendix A - Troubleshooting:

Appendix A - Troubleshooting
----------------------------

This chapter describes common issues you may encounter when working with Virtuozzo Storage clusters and ways to resolve them. The main tool you use to solve cluster problems and detect hardware failures is ``vstorage top``.

.. _Submitting Problem Reports to Technical Support:

Submitting Problem Reports to Technical Support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If your cluster is experiencing a problem that you cannot solve on your own, you can use the ``vstorage make-report`` command to compile a detailed report about the cluster. You can then send the report to the support team who will closely examine your problem and make their best to solve it as quickly as possible.

To generate a report:

1. Configure passwordless SSH access for the ``root`` user from the server where you plan to run the ``vstorage make-report`` command to all servers that participate in the cluster.

   The easiest way to do this is to create an SSH key with ``ssh-keygen`` and use ``ssh-copy-id`` to configure all servers to trust this key. For details, see the man pages for ``ssh-keygen`` and ``ssh-copy-id``.

2. Run the ``vstorage make-report`` command to compile the report:

   ::

       # vstorage -c stor1 make-report
       The report is created and saved to vstorage-report-20121023-90630.tgz

   The command collects cluster-related information on all servers participating in the cluster and saves it to a file in your current working directory. You can find the exact file name by checking the ``vstorage`` output ( ``vstorage-report-20121023-90630.tgz`` in the example above).

   If necessary, you can save the report to a file with a custom name and put it to a custom location. To do this, pass the ``-f`` option to the command and specify the desired file name (with the ``.tgz`` extension) and location, for example:

   ::

       # vstorage -c stor1 make-report -f /home/reportSTOR1.tgz

Once it is ready, submit the report to the support team. For example, you can do this using the Support Request Tracker tool. For more information on using this tool, consult the *Virtuozzo 7 User's Guide*.

.. note:: The report contains only information related to your cluster settings and configuration. It does not contain any private information.

.. _Out of Disk Space:

Out of Disk Space
~~~~~~~~~~~~~~~~~

When very little free disk space remains in a Virtuozzo Storage cluster, it is critically important to increase it as soon as possible by adding more chunk servers or removing some data. As soon as 95% of cluster disk space becomes occupied, the allocation of new data chunks is no longer possible and such requests are blocked until the cluster can satisfy the demand. As a result, user I/O becomes blocked as well, effectively freezing Containers and virtual machines.

.. note:: It is highly recommended to keep at least 10% of disk space free for recovery in case of host machine failures. You should also monitor usage history, for example, using the ``vstorage top`` or ``vstorage get-event`` commands (for more information, see :ref:`Monitoring Virtuozzo Storage Clusters`).

.. _Symptoms:

Symptoms
********

1. Stuck I/O or unresponsive mount point, ``dmesg`` messages about stuck I/O, frozen Containers and virtual machines.

2. ``vstorage top`` and ``vstorage get-event`` show error messages like "Failed to allocate X replicas at tier Y since only Z chunk servers are available for allocation".

.. _Solutions:

Solutions
*********

1. Remove any unnecessary data to free disk space.

   .. note:: Additional effect which may surprise at first is that as soon as I/O queues in the kernel are full with the blocked I/O, a mount point on the client machine may stuck responding altogether and no longer be able to service the requests even such as file listing. In this case an additional mount point can be created to list, access and remove the unneeded data.

2. Add more Chunk Servers on unused disks (see :ref:`Setting Up Chunk Servers`).

If the solutions above are not possible, you can use one of the following *temporary* workarounds:

1. Lower the replication factor for some of the least critical user data (see :ref:`Configuring Replication Parameters`). Remember to revert the changes afterwards.

2. Reduce the allocation reserve. For example, for cluster ``stor1``:

   ::

       # vstorage -c stor1 set-config mds.alloc.fill_margin=2

   where ``mds.alloc.fill_margin`` is the percentage of reserved disk space for CS operation needs (the default value is 5). Remember to revert the changes afterwards.

.. _Poor Write Performance:

Poor Write Performance
~~~~~~~~~~~~~~~~~~~~~~

Some network adapters, like RTL8111/8168B, are known to fail to deliver full-bandwidth, full-duplex network traffic. This can result in poor write performance.

So before deploying a Virtuozzo Storage cluster, you are highly recommended to test networks for full-duplex support. You can use the ``netperf`` utility to simultaneously generate in and out traffic. For example, in 1 GbE networks, it should constantly deliver about 2 Gbit/s of total traffic (1 Gbit/s for incoming and 1 Gbit/s for outgoing traffic).

.. _Poor Disk I/O Performance:

Poor Disk I/O Performance
~~~~~~~~~~~~~~~~~~~~~~~~~

In most BIOS setups, AHCI mode is configured to work by default with the **Legacy** option enabled. With this option, your servers work with SATA devices via the legacy IDE mode, which may affect the cluster performance, making it up to 2 times slower than expected. You can check that the option is currently enabled by running the ``hdparm`` command, for example:

::

    # hdparm -i /dev/sda
    ...
     PIO modes:  pio0 pio1 pio2 pio3 *pio4
     DMA modes:  mdma0 mdma1 mdma2
     UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 udma6

The asterisk before ``pio4`` in the **PIO modes** field indicates that your hard drive ``/dev/sda`` is currently operating in the legacy mode.

To solve this problem and maximize the cluster performance, always enable the **AHCI** option in your BIOS settings.

.. _Hardware RAID Controller and Disk Write Caches:

Hardware RAID Controller and Disk Write Caches
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

It is important that all hard disk drives obey the "flush" command and write their caches before the command completes. Unfortunately, not all hardware RAID controllers and drives do this, which may lead to data inconsistencies and file system corruptions on a power failure.

Some 3ware RAID controllers do not disable disk write caches and do not send "flush" commands to disk drives. As a result, the file system may sometimes become corrupted even though the RAID controller itself has a battery. To solve this problem, disable writes caches on all disk drives in the RAID.

Also, make sure that your configuration is thoroughly tested for consistency before deploying a Virtuozzo Storage cluster. For information on how you can do this, see :ref:`SSD Drives Ignore Disk Flushes`.

.. _SSD Drives Ignore Disk Flushes:

SSD Drives Ignore Disk Flushes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A lot of desktop-grade SSD drives can ignore disk flushes and fool operating systems by reporting that data was written while it was actually not. Examples of such drives are OCZ Vertex 3 and Intel X25-E, X-25-M G2 that are known to be unsafe on data commits. Such disk drives should not be used with databases and may easily corrupt the file system on a power failure.

The 3rd generation Intel SSD drives (S3700 and 710 series) do not have these problems, featuring a capacitor to provide a battery backup for flushing the drive cache when the power goes out.

Use SSD drives with care and only when you are sure that drives are server-grade drives and obey "flush" rules. For more information on this issue, read the following article about PostreSQL: http://www.postgresql.org/docs/current/static/wal-reliability.html.

.. _Cluster Cannot Create Enough Replicas:

Cluster Cannot Create Enough Replicas
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sometimes, the cluster might not create the required number of data chunks even if enough chunk servers are present in the cluster.

This may be the case when you create new chunk servers by making copies of an existing chunk server (e.g., you set up a chunk server in a virtual machine and then clone this machine). In this case, all copied chunk servers have the same UUID — that is, the UUID of the original server. The cluster has information that all chunk servers are located on the original host and cannot allocate new data chunks.

To solve the problem, generate a new UUID for a cloned chunk server by running the following command on the destination host:

::

    # /usr/bin/uuidgen -r | tr '-' ' ' | awk '{print $1$2$3}' > /etc/vstorage/host_id

For more information on the ``uuidgen`` utility, see its man page.

.. _Failed Chunk Servers:

Failed Chunk Servers
~~~~~~~~~~~~~~~~~~~~

If a chunk server in your Virtuozzo Storage cluster fails, you need to identify the cause of failure and choose a correct way to solve the problem.

Do the following:

1. Run the ``vstorage top`` command. For example:

   ::

       # vstorage -c stor1 top

2. Press **i** to cycle to the ``FLAGS`` column in the chunk server section and find the flags corresponding to the failed CS.

3. Find the shown flags in the table below to identify the cause of failure and the way to solve the problem.

+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Flag                  | Issue                                                                                                                                    | What to do                                                                                                                                                                               |
+=======================+==========================================================================================================================================+==========================================================================================================================================================================================+
| H                     | An I/O error.                                                                                                                            | Check the disk for errors. If the disk is broken, replace it and recreate the CS as described in :ref:`Replacing Disks used as Chunk Servers`. Otherwise, contact technical support.     |
|                       |                                                                                                                                          |                                                                                                                                                                                          |
|                       | The disk on which the chunk server runs is broken.                                                                                       |                                                                                                                                                                                          |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| h                     | A chunk checksum mismatch.                                                                                                               | Check the disk for errors. If the disk is broken, replace it and recreate the CS as described in :ref:`Replacing Disks used as Chunk Servers`. Otherwise, contact technical support.     |
|                       |                                                                                                                                          |                                                                                                                                                                                          |
|                       | Either the chunk is corrupt or the disk where the chunk is stored is broken.                                                             |                                                                                                                                                                                          |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| S                     | The CS journal stored on a journaling SSD is not accessible.                                                                             | Check the journaling SSD for errors. If the disk is broken, replace it as described in :ref:`Failed Write Journaling SSDs`                                                               |
|                       |                                                                                                                                          |                                                                                                                                                                                          |
|                       | Either the journal is corrupt or the journaling SSD is broken.                                                                           |                                                                                                                                                                                          |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| s                     | The chunks' checksums stored on a caching SSD are not accessible.                                                                        | Check the caching SSD for errors. If the disk is broken, replace it as described in :ref:`Failed Data Caching SSDs`                                                                      |
|                       |                                                                                                                                          |                                                                                                                                                                                          |
|                       | Either the checksums are corrupt or the caching SSD is broken.                                                                           |                                                                                                                                                                                          |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| R                     | The path to the chunk repository is invalid on CS start.                                                                                 | Make sure the disk is attached and correctly mounted. Make sure the disk's entry in /etc/fstab is correct.                                                                               |
|                       |                                                                                                                                          |                                                                                                                                                                                          |
|                       | The disk on which the chunk server runs is not attached or mounted.                                                                      |                                                                                                                                                                                          |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| T                     | An I/O request timeout.                                                                                                                  | Make sure the disk is attached and check dmesg output for I/O request timeout messages to find out why the disk might be inaccessible.                                                   |
|                       |                                                                                                                                          |                                                                                                                                                                                          |
|                       | The disk may only be inaccessible for some reason and not necessarily broken.                                                            |                                                                                                                                                                                          |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

.. _Replacing Disks Used as Chunk Servers:

Replacing Disks Used as Chunk Servers
*************************************

To replace a broken HDD or SSD disk used as a chunk server, do the following:

1. Remove the failed CS from the cluster as described in :ref:`Removing Chunk Servers`.

   .. note:: Do not detach the broken disk until you remove the CS.

2. Replace the broken disk with a new one.

3. Prepare the SSD as described in :ref:`Preparing Disks for Virtuozzo Storage`.

4. Create a new CS as described in :ref:`Stage 2: Creating a Chunk Server`.

.. _Failed Write Journaling SSDs:

Failed Write Journaling SSDs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If the SSD used to store write journals breaks, all chunk servers which had journals on this SSD will fail. The cluster will continue to work and will create new replicas to make up for those which have been lost. If you need to set up same write journals on a replacement SSD, do the following:

1. Remove the failed chunk servers as described in :ref:`Removing Chunk Servers`.

2. Prepare the SSD as described in :ref:`Preparing Disks for Virtuozzo Storage`.

3. Create new chunk servers which will keep write journals on the new SSD as described in :ref:`Configuring SSD Drives for Write Journaling`.

.. _Failed Data Caching SSDs:

Failed Data Caching SSDs
~~~~~~~~~~~~~~~~~~~~~~~~

If an SSD disk used to store read cache breaks, the cache will be lost and the client will continue to work without it. If you need to set up the same read cache on a replacement SSD, do the following:

1. Stop all virtual environments running on the client or migrate them to a different Node.

2. Unmount the cluster. For example:

   ::

       # umount /vstorage/stor1

3. Prepare the SSD as described in :ref:`Preparing Disks for Virtuozzo Storage`.

4. Create the cache and mount the cluster as you have had as described in :ref:`Configuring SSD Drives for Data Caching`.

.. _Failed MDS Servers:

Failed MDS Servers
~~~~~~~~~~~~~~~~~~

If the disk hosting an MDS server fails, replace it as follows:

1. Delete the failed MDS server as described in :ref:`Removing MDS Servers`.

2. Create a new MDS server as described in :ref:`Adding MDS Servers`.
