tests: use consistent environment variables for setting charset

Character set in POSIX is set by the locale defined (in decreasing order
of precedence) by the LC_ALL, LC_CTYPE and LANG environment variables (I
believe CHARSET is only historic). LC_ALL is cleared to ensure that
LC_CTYPE takes effect, but LC_ALL is not used to set the locale to
ensure that other parts of the locale aren't overriden, if set.  Since
there doesn't seem to be a cross-platform way of specifying a UTF-8
locale, and not all systems may support UTF-8, a <precheck> is used
(where relevant) to skip the test if UTF-8 isn't in use.  Test 1035 was
also converted to UTF-8 for consistency, as the actual character set
used there is irrelevant to the test.

tests/data/test1034

@@ -33,7 +33,12 @@ idn
 </features>
 <setenv>
 CHARSET=UTF-8
+LC_ALL=
+LC_CTYPE=UTF-8
 </setenv>
+<precheck>
+perl -MI18N::Langinfo=langinfo,CODESET -e 'die "Needs a UTF-8 locale" if (lc(langinfo(CODESET())) ne "utf-8");'
+</precheck>
  <name>
 HTTP over proxy with malformatted IDN host name
  </name>

tests/data/test1035

@@ -33,13 +33,17 @@ idn
 </features>
 <setenv>
 CHARSET=UTF-8
-LANG=en_US.UTF-8
+LC_ALL=
+LC_CTYPE=UTF-8
 </setenv>
+<precheck>
+perl -MI18N::Langinfo=langinfo,CODESET -e 'die "Needs a UTF-8 locale" if (lc(langinfo(CODESET())) ne "utf-8");'
+</precheck>
  <name>
 HTTP over proxy with too long IDN host name
  </name>
  <command>
-http://too-long-IDN-name-cürl-rüles-la-la-la-dee-da-flooby-nooby.local/page/1035 -x %HOSTIP:%HTTPPORT
+http://too-long-IDN-name-cürl-rüles-la-la-la-dee-da-flooby-nooby.local/page/1035 -x %HOSTIP:%HTTPPORT
 </command>
 </client>
 
@@ -50,8 +54,8 @@ http://too-long-IDN-name-c
 ^User-Agent:.*
 </strip>
 <protocol>
-GET http://too-long-IDN-name-cürl-rüles-la-la-la-dee-da-flooby-nooby.local/page/1035 HTTP/1.1
+GET http://too-long-IDN-name-cürl-rüles-la-la-la-dee-da-flooby-nooby.local/page/1035 HTTP/1.1
-Host: too-long-IDN-name-cürl-rüles-la-la-la-dee-da-flooby-nooby.local
+Host: too-long-IDN-name-cürl-rüles-la-la-la-dee-da-flooby-nooby.local
 Accept: */*
 Proxy-Connection: Keep-Alive
 

tests/data/test165

@@ -32,8 +32,12 @@ idn
 </features>
 <setenv>
 CHARSET=UTF-8
-LANG=en_US.UTF-8
+LC_ALL=
+LC_CTYPE=UTF-8
 </setenv>
+<precheck>
+perl -MI18N::Langinfo=langinfo,CODESET -e 'die "Needs a UTF-8 locale" if (lc(langinfo(CODESET())) ne "utf-8");'
+</precheck>
  <name>
 HTTP over proxy with IDN host name
  </name>

tests/data/test2046

@@ -42,8 +42,12 @@ idn
 </features>
 <setenv>
 CHARSET=UTF-8
-LANG=en_US.UTF-8
+LC_ALL=
+LC_CTYPE=UTF-8
 </setenv>
+<precheck>
+perl -MI18N::Langinfo=langinfo,CODESET -e 'die "Needs a UTF-8 locale" if (lc(langinfo(CODESET())) ne "utf-8");'
+</precheck>
  <name>
 Connection re-use with IDN host name
  </name>

tests/data/test2047

@@ -43,8 +43,12 @@ idn
 </features>
 <setenv>
 CHARSET=UTF-8
-LANG=en_US.UTF-8
+LC_ALL=
+LC_CTYPE=UTF-8
 </setenv>
+<precheck>
+perl -MI18N::Langinfo=langinfo,CODESET -e 'die "Needs a UTF-8 locale" if (lc(langinfo(CODESET())) ne "utf-8");'
+</precheck>
  <name>
 Connection re-use with IDN host name over HTTP proxy
  </name>