Sat Sep 3 01:03:50 AM JST 2022

Setting grub to boot with UUID

Booting with grub using UUID

Sometimes linux systems will switch the boot device corresponding to sda at boot. This simple procedure shows how to boot from a disk using a fixed UUID in order to always boot from the same device.

First, determine the device UUID with blkid.

Next, edit /etc/default/grub, adding the following line at the start with UUID information

GRUB_DEVICE_UUID="xxxx-xxxx-xxxx-xxxxxxx"

Lastly, update grub with

update-grub

Posted by Gernot Hassenpflug | Permanent link

Wed 09 Sep 2020 10:21:29 PM JST

Stopping tracker miner in GNOME

Stopping Gnome tracker miner

Anyone using a GNOME-based linux desktop environment, including MATE as do I, may have noticed vast CPU resources being dedicated to the tracker-miner-fs process at one time or other. Notice meaning, interfering with work or pleasure. It is very very hard to stop this process from interfering, and impossible to remove completely cleanly. There are many options one can try to adjust, such as expluding directories and file types, but for myself that is way too much work for a completely useless process foisted on me unwillingly. I really like MATE overall, otherwise a sane alternative might be to simply switch to Xcfe as a desktop environment instead.

So here is the solution (applicable in September 2020) to stop the filesystem miner from starting up:

gsettings set org.freedesktop.Tracker.Miner.Files enable-monitors false
gsettings set org.freedesktop.Tracker.Miner.Files crawling-interval -2
gsettings set org.freedesktop.Tracker.Miner.Files ignored-files "['*']"
pkill tracker
rm -rf ~/.cache/tracker

Apparently a reboot might be necessary.

References

Posted by Gernot Hassenpflug | Permanent link

Wed 01 Jul 2020 10:11:48 PM JST

Certbot Timeout

Certbot Renewal Timeout

On my sites with Lets Encrypt, certbot is supposed to automatically update every three months. However, almsot every time there is some problem or other. This time it was a timeout at the server:

Attempting to renew cert (aikishugyo.no-ip.org) from /etc/letsencrypt/renewal/aikishugyo.no-ip.org.conf produced an unexpected error: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45). Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/aikishugyo.no-ip.org/fullchain.pem (failure)

A search revealed the LetsEncrypt had changed over to using a different CDN provider, and it was often necessary to reduce the network card MTU, to either 1300, or even 1100.

In my case, it turned out 1300 did not work, but 1100 did:

  ifconfig enp2s0 mtu 1100

So then certbot worked again fine.

References

Posted by Gernot Hassenpflug | Permanent link

Thu 21 May 2020 11:32:48 PM JST

RDP with Linux

Running RDP from the desktop environment in linux results in something like vinagre, various variations on vnc, or rdesktop being run—with defaults! These defaults often include a 800x600 window, which is less than suitable in most cases these days, although it has the advantage of being fast in comparison with larger windows sizes. I will use rdesktop in the current example, and get back to experimenting with other clients at a later stage.

Running rdesktop with arguments can of course be done from the command line, and this is certainly a quick and simple solution. See man rdesktop for options. For example:

  rdesktop -g -u USERID 1280x1024 10.110.0.148

where USERID is the user intended to log in with on the destination machine, which can be specified either by IP address or by full-qualified host name (FQDN).

If a domain is required, the default domain will be that of the destinatiom machine. So if that machine is actually connected to an AD server, and the AD server's domain should be used instead, the is the way the used ID can be specified, as can password:

  rdesktop -u USERID -d DOMAIN -p PASSWORD HOST/IP

Some ways to save bandwith include switching off mouse movement transmission (see man page), and a couple of the below parameters, although I could not tell whether they made a difference for me:

  • -P — cache bitmaps to disk
  • -B — use Xserver backing store instead of built-in rdesktop one.
  • -a 8 — use 8-bit (or 15, 16, 24, or 32) colour depth
  • -z — compress the transmission

A last fun parameter is -T TITLE where TITLE is the desired window title.

Configuration File

After the above, one might think to put these parameters into a configuration file to use with an rdesktop session each time around, preferably from the menu.

The bad news is... there is no such configuration file. The work-around to this is to define a local file re-defining rdesktop as the command required, for example:

  rdesktop = "rdesktop -u USERID -p PASSWORD HOST/IP"

or to write this into a file, then chmod +x this file and execute it to get the desired result.

References

Posted by Gernot Hassenpflug | Permanent link

Sun 17 May 2020 02:18:13 AM JST

IBM X31 Debian runs with LXDE but not GNOME/MATE

My IBM X31 is a useful tool for ALPS printer connection. It has a parallel port, runs Windows 2000 Professional with the latest 32-bit ALPS drivers, and I have installed Debian GNU/Linux on a separate partition as well for testing of the linux driver, and forseeably–together with Windows–SCSI connections through the CardBus.

As I had not upgraded it recently, I was surprised to discover that GNOME/MATE no longer worked: the lightdm login would succeed, the UIM applet would show up, but after some delay the X server would crash and the login screen would reappear.

Another problem was that with kernels newer than 4.19 (specifically I could only confirm 5.6, I forget what previous 5.x displayed the same issue, perhaps 5.4), the display shows corruption in the form of hundreds of short cursor-type thin horizontal lines all over, randomly but clearly arranged in vertical colums all across the screen. So for all the below, I am sticking the kernel 4.19 even on my Debian sid.

Wary of graphics card issues, I spent a week and more invetigating the Radeon Modile M6 LY chip, its kernel support with the radeon module, DRI2 and later support in Mesa, and its settings under X [1]. Despite no X errors and the login screen working fine, the desktop refused to appear, with cryptic errors in the logs [2,3].

It turned out that the issue is one of CPU instruction support: specifically, SSE2 opcodes that are used in some GNOME applets at least, and which the Pentium M on the X31 does not suppot [2]. It turns out that, since at least the middle of 2019, the relevant GNOME developers have not managed to find a way to cross-compile their applications in such a way as to provide packages for the i686 acrchitecture which do not contain SSE2 opcodes [2].

The workaround, it appears, is to use LXDE, Openbox, or anything else that does not depend on GNOME [2]. Thus, I installed task-lxde-desktop and removed all MATE and GNOME packages. Voila, perfectly working desktop again!

Graphics Card Settings
/etc/initramfs-update/modules

I wanted to use the radeonfb module for the framebuffer. This is not loaded by default, so I added it to the modules file:

  1. radeonfb
  2. radeon

and then ran update-initramfs -u -k all.

/etc/X11/xorg.conf settings

The latest radeon kernel module and X server use KMS (kernel mode settings) have different parameters to set than the old UMS (user mode settings), so here is my modern KMS-capable setting.

Section "Module"
  Load "dri"
  Load "GLcore"
  Load "glx"
EndSection

/../

Section "Device"
  Identifier "Card0"
  Driver     "radeon"
  BusID      "PCI:1:0:0"
  Option     "RenderAccel"     "True"
  Option     "DRI3"            "True"
  Option     "AccelMethod"     "EXA"
  Option     "EnablePageFlip"  "True"
  Option     "SWcursor"        "True"
  Option     "ColorTiling"     "True"
  Option     "SwapBuffersWait" "True"
  # The below are not driver options anymore I think, but I left them in
  Option     "DRI"             "3"
  Option     "TearFree"        "True"
  EndSection
References
  1. Xorg ATI radeon video driver manual page:
    https://www.x.org/releases/current/doc/man/man4/radeon.4.xhtml
  2. LXDE workaround for mate-panel segfault because of SSE2 opcodes:
    https://bbs.archlinux32.org/viewtopic.php?id=2803
  3. Another case of crashing panel:
    https://forums.linuxmint.com/viewtopic.php?t=300452

Posted by Gernot Hassenpflug | Permanent link

Sat 16 May 2020 03:04:52 AM JST

Age check for purchase in Zen Cart

Introduction

In Zen Cart, we need to add an age check for purchases, since our shop sells alcohol. Japanese regulations specify that the age check must be a text box for users to type in their age in digits, not a scrollbar and no autofill. And that this check must be there regardless of whether user registration is limited to users above age 20 (the legal age for purchase of alcohol in Japan).

In additions, notice of the age restriction must be included on customer emails pertaining to the order, and on the invoice and packing slip (deliveries must be accepted by a person above the legal limit for purchase of alcohol).

So here is what I have done, in case it helps others to do something similar, or elicits feedback on improvements. The age check is implemented in the conditions section of the checkout_payment page for standard 3-page checkout process, or on the checkout_one page if using OPC (One Page Checkout plugin). Additional work is needed for additing warning notices to invoice and packaging slip, this is not covered here.

If conditions permitted, the age check condition could go on the checkout_shipping page, displayed with tpl_checkout_shipping_default.php. This page reloads itself, so the validation would also be included directly on this page, rather than editing the checkout_confirmation page.

Standard Checkout versus OPC

For standard 3-page checkout, the checkout_process flow goes like this:

  1. shopping_cart
  2. checkout_shipping [Step 1/3]
  3. checkout_payment [Step 2/3] (age check added to conditions)
  4. checkout_confirmation [Step 3/3] (age check validation here)

For One Page Checkout (OPC), the checkout_process flow goes like this:

  1. shopping_cart
  2. checkout_one (age check added to conditions)
  3. checkout_one_confirmation(age check validation here)
Step1

Add age_check to table orders in your database:

ALTER TABLE orders ADD order_age_check int(3) NOT NULL;
Step 2

Prepare some language definitions for wording used in the age check conditions.

First, in your override language checkout_payment.php file
includes/languages/LANGUAGE/YOURTEMPLATE/checkout_payment.php for the box in the display:

define('TEXT_AGE_CHECK_DESCRIPTION','The Japanese tax office requires confirmation of legal age for the purchase of alcohol, by entering your age when confirming your purchase.');
define('TEXT_AGE_CHECK','Enter your age in English numerals.');

Note, if using OPC, add this to your checkout_one.php file.

Second, in your override general language file
includes/languages/YOURTEMPLATE/LANGUAGE.php add definitions for validation errors:

define('ERROR_AGE_CHECK_INPUT', 'You must be over 20 years of age to purchase alcohol from our store.');
define('ERROR_AGE_CHECK_INPUT_INHUMAN', 'We do not expect humans over this age.');

Third, to enable outputing a notice of age restriction in emails, and on the order invoice/packing slip, set definitions in
languages/LANGUAGE/YOURTEMPLATE/checkout_process.php as follows:

define('EMAIL_AGE_CHECK_HEADER','Restriction:');
define('EMAIL_AGE_CHECK','Sales and delivery of alcohol is prohibited by law to persons under 20 years of age.');
Step 3

Add use of the age_check fields to your includes/classes/order.php. Note this is a core file edit.

1. function query($order_id)
   $this->info = array(...
              'age_check' => $order->fields['order_age_check'], // age check

2. function cart()
   $this->info = array(...
              'age_check' => $_SESSION['age_check'], // age check

3. function create($zf_ot_modules, $zf_mode = 2)
   $sql_data_array = array(...
                  'order_age_check' => $this->info['age_check'], // age check

4. $email_order ...
   $email_order .= "\n" . EMAIL_AGE_CHECK_HEADER . "\n" .
   ...
   EMAIL_AGE_CHECK . "\n";

The first three are additions to arrays, the fourth an output for the email being sent.

Step 4

Add processing of the age check to your checkout_payment page as follows.
Note these are core file edits (I am not sure if overrides for individual pages are possible).

First, add the following to
includes/modules/pages/checkout_payment/header_php.php:

// age check
if (isset($_SESSION['age_check'])) {
    $age_check = $_SESSION['age_check'];
}

Second, add the following to
includes/modules/pages/checkout_payment/jscript_main.js:

window.onload = function resetAgeCheck()
{
    //var x=document.getElementsByTagName("input");
    var x = document.getElementById("age_check");
    x.value = "";
}

The code is there to check whether the check has already been set, if the user is returned to this page by another error.
Note, if using OPC, add this to your checkout_one page header file along with other customizations as described in the OPC instructions.

Step 5

Add validation to your checkout_confirmation page as follows.
Note these are core file edits (I am not sure if overrides for individual pages are possible).

In the header file for the page,
includes/modules/pages/checkout_confirmation/header_php.php, add the input check to the age check box is digits and within acceptable ranges (above minimum, below silly).

// age check
if (zen_not_null($_POST['age_check'])) {
    $_SESSION['age_check'] = zen_db_prepare_input($_POST['age_check']);
}
$age_check = $_SESSION['age_check'];
...
// age check required even if not null
$min_age=20; // legal age for purchase and consumption of alcohol (Japan)
$max_age=122; // oldest human known!
if (filter_var($age_check, FILTER_VALIDATE_INT, array("options" => array("min_range"=>$min_age, "max_range"=>$max_age))) === false) {
    if ($age_check <= $max_age) {
        $messageStack->add_session('checkout_payment', ERROR_AGE_CHECK_INPUT,'error');
    } else {
        $messageStack->add_session('checkout_payment', ERROR_AGE_CHECK_INPUT_INHUMAN,'error');
    }
}

Note, if using OPC, add this to your checkout_one_confirmation page header file.

Step 6

Add the age check box to the conditions section of your override template file
includes/templates/YOURTEMPLATE/templates/tpl_checkout_payment_default.php so that is appears something like this:

<?php
  if (DISPLAY_CONDITIONS_ON_CHECKOUT == 'true') {
?>
<fieldset>
<legend><?php echo TABLE_HEADING_CONDITIONS; ?></legend>
<div><?php echo TEXT_CONDITIONS_DESCRIPTION;?></div>
<?php echo  zen_draw_checkbox_field('conditions', '1', false, 'id="conditions"');?>
<label class="checkboxLabel" for="conditions"><?php echo TEXT_CONDITIONS_CONFIRM; ?></label>
<!-- Age verification input field as required by Japanese tax office -->
<div><?php echo TEXT_AGE_CHECK_DESCRIPTION;?></div>
<?php echo  zen_draw_input_field('age_check', '', 'id="age_check" pattern="[0-9]*" required minlength="2" maxlength="3" size="3" placeholder="00"', 'text');?>
<label class="inputAgeLabel" for="conditions"><?php echo TEXT_AGE_CHECK; ?></label>
</fieldset>
<?php
  }
?>

Note, if using OPC, add this to your override of tpl_modules_opc_conditions.php.

Add styling to your override stylesheet
includes/templates/YOURTEMPLATE/css/stylesheet.css:

LABEL.inputAgeLabel {width:15em;margin:0.5em 0.3em;}
input[type=text]#age_check {width:3em;margin-left:20px;font-size:120%;margin-top:5px;}
Reference

Changes and files to edit are as follows.

Tables
  1. orders
Processing Files
  1. includes/classes/order.php
Box display
  1. includes/templates/YOURTEMPLATE/templates/tpl_checkout_payment_default.php
  2. includes/templates/YOURTEMPLATE/css/stylesheet.css
OPC Box display
  1. includes/templates/YOURTEMPLATE/templates/tpl_modules_opc_conditions.php
  2. includes/templates/YOURTEMPLATE/css/stylesheet.css
Language Definitions

Note multiple languages.

  1. includes/languages/LANGUAGE/YOURTEMPLATE/checkout_process.php
  2. includes/languages/LANGUAGE/YOURTEMPLATE/checkout_payment.php
  3. includes/languages/YOURTEMPLATE/LANGUAGE.php (validaton)
OPC Language Definitions

Note multiple languages.

  1. includes/languages/LANGUAGE/YOURTEMPLATE/checkout_process.php
  2. includes/languages/LANGUAGE/YOURTEMPLATE/checkout_one.php
  3. includes/languages/YOURTEMPLATE/LANGUAGE.php (validaton)
Page Edits
  1. includes/modules/pages/checkout_payment/header_php.php
  2. includes/modules/pages/checkout_payment/jscript_main.js
  3. includes/modules/pages/checkout_confirmation/header_php.php (validation)
OPC Page Edits
  1. includes/modules/pages/checkout_one/header_php.php
  2. includes/modules/pages/checkout_one/jscript_main.js
  3. includes/modules/pages/checkout_one_confirmation/header_php.php (validation)

Posted by Gernot Hassenpflug | Permanent link

Wed 15 Apr 2020 06:28:50 PM JST

Fixing /var/lib/dpkg/available

I finally fixed an old but very durable Celeron-powered Toshiba DB65C/4RC, which culminated in mating the top with the bottom of a DB64P/4RC, thereby upgrading to a Pentium III.

This machine runs Microsoft Windows 2000 Professional, and I use it to connect my ALPS printers, which work best with a native parallel port.

I also have Debian installed as a dual-boot option. I found that I needed to upgrade the OS version from the old Debian Squeeze mixed with some Lenny, through Wheezy, Jessie, Stretch, and finally to Buster. Try that with Windows!

During the upgrade, I found that I had some self-compiled kernel packages in the package cache which no amount of aptitude or apt-get commands would remove. Finally, I discovered that the following file needed to be recreated:

/var/lib/dpkg/available

This can be done with:

dselect update
Reference

https://lists.debian.org/debian-user/2002/10/msg06480.html


Posted by Gernot Hassenpflug | Permanent link

Mon 06 Apr 2020 02:38:47 JST

Training in Japan during COVID-19

It has been a long time since I blogged about training Aunkai in Japan. Finally I have the motivation, time, and confidence to return to writing about my own training.

COVID-19 has seen Japan in a situation of reduced activity. Increased working from home, closure of clubs and curtailing of training activities, anything to reduce close physical contact between large numbers of people.

In that climate, I am still attending class on Thursdays and Saturdays. We all wear masks, try to slow the tempo of training so that we do not breathe heavily, and avoid close contact in exercises and application training. We also take regular breaks to wash hands, rinse mouths, and slow down.

If the situation worsens, which is quite likely seeing as how the number of infected persons in Tokyo is increasing daily, training may be cancelled, but until then. we do the best we can. I am also going to the venue by scooter, rather than taking the train, to further reduce my exposure.

UPDATE 2020-04-09: Following the government's nation-wide semi-lockdown announcement on 2020-04-07, the training venue has been closed until 2020-05-09.
Focus

The situation presents a great opportunity to focus on the aspects of training that do not depend on strength, speed, size, or aggression.

During warm-up training I can focus on keeping the diaphragm suspended by keeping the back near the lower part of the shoulder blades up, while sinking the lower back, thereby keeping space in the abdomen area for movement to occur.

Keeping the nape of the neck back then allows a slide of weight down an angled line from the back of the neck to the front of the abdomen to start off movements, or to absorb incoming forces without leaning or resisting at the point of contact.

Balancing on the heels for instilling a sense of imbalance, as opposed to a solid stance, is great for training sensitivity and timing, so that contact with the partner and application of weight is achieved before any sense of tension of resistance occurs.

During shiko and mabu and tenchijin exercises the increased space in the abdominal and chest area can be tied to the movement of the arms, and the balance on the heels moved to be more of a balance on the kua region.

Finally, as far as resistance, tension, core is concerned: any force taken in the hands can be focussed to the connection of the back, near the lower shoulder blade area, and connected to the downward slide, while the body stays square in a plane facing forward, using only motion in the sagittal plane, no rotation or twisting side to side. In other words, arms move as though pushing a stick or sword forward along the centerline perpendicular to the front of the body. Any other movements then become an add-on to this.


Posted by Gernot Hassenpflug | Permanent link | File under: BodyWork

Thu 02 Apr 2020 23:19:03 JST

Importing VMWare VM to VirtualBox and correcting disk geometry

Fixing disk geometry for "error loading operating system" (Windows XP from VMWare)

I hope this post can help others importing working VMs (not only Windows XP) from VMWare Workstation/Player to VirtualBox (converted with ovftool), suffering from "error loading operating system", where the original VMWare VMs work fine in VMWare Workstation/Player.

Posted on the VirtualBox forums:
https://forums.virtualbox.org/viewtopic.php?f=2&t=97511

Symptom:

VMWare Workstation/Player Windows XP virtual machine converted to OVF and imported into VirtualBox does not boot, displaying "error loading operating system" on black screen.

Summary:

VirtualBox does not correctly set the disk geometry in certain imported VDI files.

A simple fix to the VirtualBox VDI disk geometry solves the problem, using CHS data from original VMDK file:

vbox-img geometry --filename <diskimage>.vdi --format VDI --cylinders <C> --heads <H> --sectors <S>

Checks:

  • No obvious issues with disk controller setting or MBR.
  • Several threads (listed at end) referenced.
    Many dealt with issues related to P2V process.
    Common issues were Windows XP drivers and MBR.
    These are not applicable in this case.
  • Several unsolved threads seem to be like this case.

Solution:

  • One or two of the referenced threads mentioned disk geometry issues in converted VirtualBox virtual machines.
    VirtualBox conversion of VDMK disk during OVF import does not correctly set the CHS (cylinder/head/sector) values in the VDI file for the imported VMDK disk under certain conditions. I think from reading the various threads that the developers have stated that only standard geometry of 255 heads and 63 sectors is expected, and that the VMDK CHS information is ignored. When this expectation is not met, the VDI file will have the wrong information.
  • Check the original VMDK and converted VDI disk geometry in VBox.log.
  • Manually set correct geometry using vbox-img command.

Details:

Two original VMWare VMs, both clones of the same Windows XP Home VM, using VMWare, probably VMWare Workstation 9.x on Debian GNU/linux 32-bit or 64-bit, sometime around 2010-2014.

Current hardware (Intel Core2Duo motherboard) too old for latest VMWare Workstation, while Debian GNU/linux kernel too new for old VMWare workstation 9 or 10 to build necessary modules.

So, I converted the VMWare virtual machines (both with VMWare workstation 9 hardware compatibility) to OVF and attempted to import into VirtualBox 6.1.4, during which the VMDK disk is imported as a VDI file (this is where the problem arises).

One of them worked fine, the other did not, with the error mentioned.

The results was the same whether performing conversion and import on my 64-bit Debian GNU/linux machine or on 64-bit Windows 10.

Despite reading of numerous threads about this error, and trying to fix from the OS level, it turned out that the problem was incompatible disk geometry in the VMDK file, and VirtualBox import does not correctly use the data in the VDI disk file.

So, the solution I found was to change the geometry of the VDI disk file.

After importing and attempting to boot the imported VMs in VirtualBox, I could see In the VBox.log file the original VMWare VMDK file CHS information and the converted result.

First (working) Windows XP Home:

Checking VMDK:
SCSI capacity: 83886080
logical sector size 512
geometry (CHS): 5221/255/63

VBox.log file after importing OVF (which converts VMDK to VDI file):
CHS = 1024/255/63
VMWare was 5221/255/63

vbox-img shows 1024/255/63 in VDI disk

vbox-img info --filename <diskimage>.vdmk
Header: Geometry PCHS=16383/16/63 LCHS=1024/255/63 cbSector=83886080

VBox.log:

00:00:01.326888 PcBios: SCSI LUN#0 LCHS=1024/255/63
00:00:01.378575 VMMDev: Guest Log: BIOS: SCSI 0-ID#0: LCHS=1024/255/63 0x0000000005000000 sectors
Second (problematic) Windows XP Home:

Checking VMDK:
SCSI capacity: 83886080
logical sector size 512
geometry (CHS): 5874/255/56

VBox.log file after importing OVF (which converts VMDK to VDI file):
CHS = 5220/255/63
VMWare was 5874/255/56

vbox-img shows 0/0/0 in VDI disk

vbox-img info --filename <diskimage>.vdmk results:
Header: Geometry PCHS=0/16/63 LCHS=0/0/0 cbSector=83886080

VBox.log:

00:00:01.191130 PcBios: SCSI LUN#0 LCHS not provided
00:00:01.230446 VMMDev: Guest Log: BIOS: SCSI 0-ID#0: LCHS=5220/255/63 0x0000000005000000 sectors
Correction

Solution was to use vbox-img, included with VirtualBox, to change the geometry of the VDI disk:

vbox-img geometry --filename <diskimage>.vdi --format VDI --cylinders 5874 --heads 255 --sectors 56

Now that this matched the geometry of the original VMDK file, Windows booted up perfectly.
VBox.log entries:

00:00:01.140001 PcBios: SCSI LUN#0 LCHS=5874/255/56
00:00:01.187537 VMMDev: Guest Log: BIOS: SCSI 0-ID#0: LCHS=1024/255/56 0x0000000005000000 sectors

I have no explanation why one clone had a geometry which was compatible with Virtualbox expectations, and the other one had a different geometry.

Referenced threads:

  1. Oct 2019: current discussion (latest discussion)
    Error loading operating System error when loading a XP Guest on a W10 Host.
    https://forums.virtualbox.org/viewtopic.php?f=6&t=95312&p=461562
  2. July 2018: no solution
    error loading operating system
    https://forums.virtualbox.org/viewtopic.php?f=1&t=88923&p=426436
  3. July 2015: no solution
    Yosemite using a bootcamp as a VM
    https://forums.virtualbox.org/viewtopic.php?f=8&t=68756&p=327375
  4. Sep 2014: same issue - led me to solution
    Geometry of imported disk images
    https://forums.virtualbox.org/viewtopic.php?f=1&t=63454&p=297969
  5. Sep 2014: possibly same issue
    Vmware vmdk to virtualbox
    https://forums.virtualbox.org/viewtopic.php?f=7&t=63861&p=300061
  6. Feb 2014: possibly similar issue
    error loading operating system XP
    https://forums.virtualbox.org/viewtopic.php?f=8&t=60208&p=280213
  7. Jan 2014: different issue
    Mac OS as host, Ubuntu 12.04 (64b) guest as dual-boot on VB
    https://forums.virtualbox.org/viewtopic.php?f=8&t=59529&p=276902
  8. July 2013: possibly different issue
    Discussion: MacOS Bootcamp Partition as a VBox Guest
    https://forums.virtualbox.org/viewtopic.php?f=8&t=48704&p=318314
  9. July 2013: different issue
    [Solved] vboxmanage - resized to 50TB instead of 50GB
    https://forums.virtualbox.org/viewtopic.php?f=1&t=56261&p=260448
  10. June 2013: same issue no doubt, discussion but no solution
    Windows XP vmdk "ERROR LOADING OPERATING SYSTEM"
    https://forums.virtualbox.org/viewtopic.php?f=2&t=56195&p=260535
  11. Jan 2013: likely same issue
    VMWare VM
    https://forums.virtualbox.org/viewtopic.php?f=6&t=53533&p=245620
  12. Dec 2012: likely same issue
    vmdk file fails to load in Vbox
    https://forums.virtualbox.org/viewtopic.php?f=6&t=52924&p=242640
  13. Aug 2012: different issue
    [FIXED] Existing Windows 7x64 machine do not run
    https://forums.virtualbox.org/viewtopic.php?f=36&t=51049&p=233826
  14. April 2011: different issue
    Raw Disk Access for existing Windows XP with host opensuse
    https://forums.virtualbox.org/viewtopic.php?f=7&t=41018&p=184445
  15. Feb 2011: could be same issue
    Using VMWare SCSI Image in VB 4.0.2
    https://forums.virtualbox.org/viewtopic.php?f=2&t=38776&p=242645
  16. Oct 2010: different issue
    "Error loading operating system" after host reinstall
    https://forums.virtualbox.org/viewtopic.php?f=2&t=35416&p=158679
  17. Mar 2010: could be same issue
    Problem using VMWare vmdk images with Virtualbox
    https://forums.virtualbox.org/viewtopic.php?f=1&t=28611&p=127642
  18. Feb 2010: possibly related
    Importing Xen virtual appliance?
    https://forums.virtualbox.org/viewtopic.php?f=1&t=28354&p=126281
  19. Aug 2009: related issues discussed here
    Howto: Windows XP in both VM and native - DISCUSSION
    https://forums.virtualbox.org/viewtopic.php?f=2&t=9700&p=95231
  20. Dec 2008: different issue
    Cant install windows xp on VM
    https://forums.virtualbox.org/viewtopic.php?f=7&t=12101&p=49311
  21. Sep 2008: different issue
    how to use vmdk files with XP host
    https://forums.virtualbox.org/viewtopic.php?f=6&t=9360&p=36143
  22. Dec 2007: migrating windows XP (different issue)
    How to disable LBA on Raw disk non-LBA partition
    https://forums.virtualbox.org/viewtopic.php?f=7&t=3408&p=11947
  23. Nov 2007: different issue
    Error loading operating system (Win XP raw guest on Gutsy)
    https://forums.virtualbox.org/viewtopic.php?f=2&t=2776&p=9734

Posted by Gernot Hassenpflug | Permanent link

Fri 08 Jul 2016 23:06:31 JST

Removing Journaling on EXT3 external USB disks to work around DPO or FUA error

Problem

On several external USB disks connected to my linux system with kernel later than 3.2, I found that the disks would not mount anymore, even though they had worked perfectly up until then. Strange errors on mounting appeared in dmesg, but the disk appeared perfectly fine when attached to a Windows XP system.

An example of dmesg (or content of /var/log/messages) on mounting such a disk (taken from the reference but exactly like my own experience):

[15943.225429] usb 2-1.2: Product: USB Mass Storage Device
[15943.225434] usb 2-1.2: Manufacturer: Myson Century, Inc.
[15943.225440] usb 2-1.2: SerialNumber: 100
[15943.226446] usb-storage 2-1.2:1.0: USB Mass Storage device detected
[15943.226592] scsi16 : usb-storage 2-1.2:1.0
[15944.230947] scsi 16:0:0:0: Direct-Access     TOSHIBA  MK1233GAS        AA01 PQ: 0 ANSI: 0 CCS
[15944.231938] sd 16:0:0:0: Attached scsi generic sg1 type 0
[15944.232514] sd 16:0:0:0: [sdb] 234441648 512-byte logical blocks: (120 GB/111 GiB)
[15944.233201] sd 16:0:0:0: [sdb] Write Protect is off
[15944.233208] sd 16:0:0:0: [sdb] Mode Sense: 00 14 00 00
[15944.233888] sd 16:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[15944.248756]  sdb: sdb1
[15944.251545] sd 16:0:0:0: [sdb] Attached SCSI disk

Analysis

It turns out after some searching [1] that it seems to be a kernel bug since 2.3.32, affecting EXT3 and EXT4 (and it seems also NTFS although I have not experienced this). Nor does it seem to affect USB sticks or SD cards. In other words, only certain USB controller chips are affected.

FUA stands for "Force Unit Access" and does exactly what it means:
it forces write requests all the way to the persistent storage—HDDs—in an immediate manner,
thus "passing by" the write cache of the device.
FUA commands are used as a part of the "write barriers" logic for some file systems, which deal - exactly, with the Journaling!

According to the above [see reference link from LWN below [2]], FUA and FLUSH commands are issued by the file system itself, but executed by the block layer.

Solution

A work-around is thus to disable journaling, as follows for example:

tune2fs -O ^has_journal /dev/sdb1

Status

A kernel bug has been filed, but since December 2015 no progress has been observed there [3]. However, a further workaround (which I have not tried out) seems to be a permanent fix for the situation:

echo 'temporary write through' > /sys/block/sdb/device/scsi_disk/*/cache_type

References

  1. Gentoo forums: Libparted warning Error fsyncing/closing...Remote I/O errror
  2. LWN: The End of Block Barriers
  3. Kernel bug report: Bug 89511 - USB-storage mount error

Posted by Gernot Hassenpflug | Permanent link