Moodle Association of Japan Discussion Forum

Hankaku and Zenkaku issue

Hankaku and Zenkaku issue

by David CAMPBELL -
Number of replies: 22

Sorry for crossposting this and this is probably not what this forum is for, but I haven't had a response on the Moodle.org Japanese language forum. I am hoping someone who has experienced the same thing can help me out.

A Japanese instructor sent me this:

"My problem on moodle is about quiz.

I always make quiz for every lectures.
In this quiz (embedded cloz), I ask the students to input their answer in HANKAKU for English and number, and  ZENKAKU for Japanese. But they dont take care about the rule.
In such case, I have to prepare more answers for each."

 How do you handle this problem? Is there something I need to do in the settings as the Administrator or is there something he can do at the course or quiz level to solve this problem?

Thanks for the help.

P.S. If you want to write the answer in Japanese that is okay.

In reply to David CAMPBELL

Re: Hankaku and Zenkaku issue

by Gordon BATESON -
Hi David,
as far as I know, there is no simple setting in Moodle to force single-byte characters in answers to certain questions in a certain quiz.

However, it may be possible to solve this problem programmatically. Moodle has a setting to allow you to insert anything you like into the <HEAD> part of all your Moodle pages. Administration -> Site administration -> Appearance -> Additional HTML. This is often used to insert GoogleAnalytics code, but you could create your own javascript and put in there. The javascript would detect when it was on a page that required single-byte input, and then would convert all double-byte letters, numbers and punctuation to their single-byte equivalents.

At JALTCALL 2007, I did a joint presentation (with Don, Andy, Peter, Paul D., et. al.) about tweaking Moodle which included a demonstration of converting double-byte to single-byte chars. You can download a sample javascript file from here:

http://englishforum.sgu.ac.jp/downloads/jaltcall2007/bateson/two-byte-utf8.js

If you give that to a programmer, it may well serve as a basis to implement what you want.

regards
Gordon
In reply to Gordon BATESON

Re: Hankaku and Zenkaku issue

by David CAMPBELL -

Thank you Gordon. I will take a look and see what happens.

Okumura-sensei replied to my question on the Japanese Moodle Forum and raised an interesting point. Do you think it would be possible to get a choice  as part of the embedded cloz that would force the students to use the correct one? Right now there way to require the case or ignore the case in a question. 

In reply to David CAMPBELL

Re: Hankaku and Zenkaku issue

by Gordon BATESON -
Hi David,
yes it would be possible, however you cannot simply tell the browser to accept only a subset of all the characters that can be typed or tapped on the computer's keyboard.

Instead you would need to add some extra programmatic script to convert double-byte chars to single-byte. This could be done either by javascript that is added to the page and converts characers as they are typed in, or by PHP on the server that converts the characters when they are received from the student's browser.

As I mentioned before, you can add the JavaScript via the Moodle interface, but to add the PHP you would need to meddle with the Moodle core PHP, which is messy and makes upgrades troublesome. Therefore, I suggest the javascript option is easier and more long-lasting, even though the PHP option will definitely cater for ALL cases, where the javascript option may not work in some situations, for example where the browser disallows javascript completely.

regards
Gordon
In reply to Gordon BATESON

Re: Hankaku and Zenkaku issue

by Jetha CHAN -

Hi Gordon,

Just thought I'd quickly chime in on behalf of Moodle HQ. This issue has just been brought to our attention by David Campbell and we're going to take a stab at it as a PHP-based solution as a part of Moodle core - we'll be able to leverage mb_convert_kana() on the server, and that'll take care of users with JS disabled as well.

Unfortunately we won't be able to support it directly for Moodle versions earlier than 2.7, as 2.6 just passed out of our regular support period ( https://docs.moodle.org/dev/Releases ) ; if a backport to earlier versions is required, contact your local Moodle partner and they should hopefully be able to sort you out.

We'll post again in this thread after we've done some exploratory work.

Best regards,

Jetha Chan

In reply to Jetha CHAN

Re: Hankaku and Zenkaku issue

by Haruhiko OKUMURA -
As Jetha Chan has pointed out, just a line of PHP code
$str = mb_convert_kana($str, "KVas");
will do the trick. wink
In reply to Jetha CHAN

Re: Hankaku and Zenkaku issue

by David CAMPBELL -

Jetha, 

Thanks for the fast response. I am sure this type of change will make a lot of instructors in Japan happy. We will be upgrading to the newer version next spring, so I am sure this instructor can deal with the situation until then.

In reply to David CAMPBELL

Re: Hankaku and Zenkaku issue

by Christian THOMPSON -

Instead of mucking about with the code, why not use the following to import the question (in GIFT format) with both possible answers: note one 4 is HANKAKU and the other is ZENKAKU?  It will take more work, but once it's done, it's done.

Two plus two equals {=4 =4}

Alternatively, instead of worrying about it, just give 0 credit for answers in the wrong format.  The students will learn quickly enough.  Technology can't solve every problem...sadly!

 

 

 

 

In reply to Christian THOMPSON

Re: Hankaku and Zenkaku issue

by David CAMPBELL -

Christian, 

That is what he is doing now, putting in both answers. I have followed the 0 points for the wrong input approach, but he doesn't seem to be willing to do that. I might get him to start by giving the partial credit when the incorrect input is used and make sure there is feedback saying why. After two or three quizzes he could switch to zero points. 

In reply to Christian THOMPSON

Re: Hankaku and Zenkaku issue

by Peter RUTHVEN-STUART -

Hello,

I tend to agree with Christian T:

> Alternatively, instead of worrying about it, just give 0 credit for answers in the wrong format.  The students will learn quickly enough.  Technology can't solve every problem...sadly!

We are often too kind to (university) students. I used to look for a technological / code solution to problems like these, but think that if at all possible it's better to get students to 'think & learn'. In this particular case, they need to know the difference between Hankaku and Zenkaku, so creating a code solution is perhaps a disservice. As long as the quizzes make clear that they need to type  in Hankaku and some of the initial questions have feedback which reminds them to type in Hankaku, then students will learn.

In reply to Peter RUTHVEN-STUART

Re: Hankaku and Zenkaku issue

by Jetha CHAN -

Hey Peter,

As a devil's advocate here - certainly students should know better, and if you're explicitly requesting hankaku or zenkaku for given situations, it's the course creator's prerogative whether to accept or handle those cases mercifully.

However, isn't it also clear that those aren't the only situations that this might apply to? I mean, let's say you had a short answer question for which the answer would be "Webサービス" - we currently can't handle a situation where someone types "Ｗｅｂサービス" with grace, and that's probably a capability that's needed.

In any case we're thinking of an admin-level setting (because it's not possible to specify options for short-answer questions in Cloze questions) that'll turn on less naive comparisons so non-Japanese users aren't affected.

Regards,

Jetha

In reply to Jetha CHAN

Re: Hankaku and Zenkaku issue

by Peter RUTHVEN-STUART -

Hello Jetha,

> However, isn't it also clear that those aren't the only situations that this might apply to? ...

Yes, and another situation would be a high-stakes test. I would not want to penalize or fail a student just because they'd typed in zenkaku instead of hankaku. So a feature to allow either zenkaku or hankaku in quizzes (i.e. auto convert one to the other) would be welcome, but even better if it could be turned off at the activity level in the same way that filters can be turned on or off at the activity level.

 

In reply to Peter RUTHVEN-STUART

Re: Hankaku and Zenkaku issue

by Jetha CHAN -

Hi Peter,

At the activity level sounds perfect; I've amended our notes on this issue accordingly, over at https://tracker.moodle.org/browse/MDL-48209.

In reply to Jetha CHAN

Re: Hankaku and Zenkaku issue

by Gordon BATESON -
Hello Jetha,
another place I often want to convert double-byte to single-byte chars is on the user profile pages.

Could we add an admin setting, perhaps on Users -> Accounts -> User Profile fields, that forcibly converts to hankaku where possible on the user profiles?

regards
Gordon
In reply to Gordon BATESON

Re: Hankaku and Zenkaku issue

by Jetha CHAN -

Hi Gordon,

As I understand it, the issue is that sometimes users input double-byte chars instead of single-byte chars in their user profile fields, and the result isn't the prettiest thing in the world.

I'm not sure how much sense that makes as core Moodle functionality (though I'll bounce it off some people here) - but it definitely does make sense as a plugin; I'll see what I can do in the next project week we have. Would something that

  1. adds the ability to flag fields that should be converted to hankaku, and
  2. retroactively converts-to-hankaku all extant field data

fit the bill?

Regards,

Jetha

In reply to Jetha CHAN

Re: Hankaku and Zenkaku issue

by Gordon BATESON -
Hi Jetha,
Yes that sounds great. The particular problem I have is with the names. If names contain a mixture of single and double byte characters, then searching and sorting becomes difficult. In some cases, I would also like to convert single byte characters to double byte equivalents in the description.

regards
Gordon
In reply to Gordon BATESON

Re: Hankaku and Zenkaku issue

by Gordon BATESON -
Hi Zetha,
one issue that is perhaps related to the profile fields is the login page. Could we convert the username/password on the login page to single-byte too?

The reason I ask is because oftentimes when I look at the report of failed login attempts, I see many instances of users having typed in the correct username, but using zenkaku instead of hankaku and the login fails. Could we fix that too?

regards
Gordon
In reply to Gordon BATESON

Re: Hankaku and Zenkaku issue

by Jetha CHAN -

Hi Gordon,

Unfortunately this isn't quite as simple as it might look. Default username policy for a Moodle site specifies that a username be only comprised of alphanumeric lowercase characters, underscores (_), hyphens (-), periods (.) or at symbols (@) alone, but by enabling the admin setting extendedusernamechars it's possible to set full-width usernames too (e.g. a user with the username ムードル博士).

In order to address the issue you've identified, we would need to detect whether extendedusernamechars is enabled. If it's disabled, we should perform a zenkaku->hankaku conversion on submitted usernames; if it's enabled, we should do nothing (or we'd stop users from being able to log in).

Will raise an issue for this once things are a little less busy here at HQ.

Cheers,

Jetha

In reply to Jetha CHAN

Re: Hankaku and Zenkaku issue

by David CAMPBELL -

I agree with Peter on this I really like when I can control things at the activity level or at the worst the course level.

In reply to David CAMPBELL

Re: Hankaku and Zenkaku issue

by Haruhiko OKUMURA -
A (tangentially) related matter:

If you use a character outside of Unicode BMP (16-bit Basic Multilingual Plane) such as emoji, "Error writing to database" results and all you wrote is lost (at least when using MySQL). It'd be nice if Moodle could filter out offending characters before database error occurs.
In reply to Haruhiko OKUMURA

Re: Hankaku and Zenkaku issue

by Jetha CHAN -

Okumura-san,

Wow, that's a really egregious bug! I don't suppose I could ask you for some more details - affected version(s), explicit problem reproduction steps? As this thread highlights, we unfortunately don't have as much visibility into Japanese issues as we would like; if you have any more information on this or any other bugs, we'd love to hear it.

We *are* thinking of implementing full emoji support in Atto, the new text editor available in 2.7+ (see https://tracker.moodle.org/browse/MDL-46779 ), but I'm not sure how long it'll be before that's done.

Best regards,

Jetha

In reply to Jetha CHAN

Re: Hankaku and Zenkaku issue

by Haruhiko OKUMURA -
Hi, Jetha,

Please follow these two steps:

1. Open http://oku.edu.mie-u.ac.jp/~okumura/temp/ and copy the notebook emoji.
2. Try to post some message to this Moodle forum with the emoji pasted.

You'll get a database error. Your message is lost even if you click the "back" button on your browser.

Emoji are beyond the Unicode 16-bit BMP, as are some kanji (e.g. the 2nd example on http://oku.edu.mie-u.ac.jp/~okumura/temp/ ('shikaru' (to scold)).
In reply to Haruhiko OKUMURA

Re: Hankaku and Zenkaku issue

by Jetha CHAN -

Okumura-san,

Thanks for that. I've opened an issue for this on our tracker ( https://tracker.moodle.org/browse/MDL-48228 ) and performed some exploratory testing; the problem presents even in the latest Moodle version when using MySQL, but not when using PostgreSQL.

Regards,

Jetha