tests: use consistent environment variables for setting charset

The character set in POSIX is set by the locale defined by (in
decreasing order of precedence) the LC_ALL, LC_CTYPE and LANG
environment variables (CHARSET was used by libidn but not libidn2).
LC_ALL is cleared to ensure that LC_CTYPE takes effect, but LC_ALL is
not used to set the locale to ensure that other parts of the locale
aren't overridden.  Since there doesn't seem to be a cross-platform way
of specifying a UTF-8 locale, and not all systems may support UTF-8, a
<precheck> is used to skip the test if UTF-8 can't be verified to be
available.  Test 1035 was also converted to UTF-8 for consistency, as
the actual character set used there is irrelevant to the test.

This patch uses a different UTF-8 locale than the last attempt, namely
en_US.UTF-8. This one has been verified on 7 different Linux and BSD
distributions and is more complete and usable than the locale UTF-8 (on
at least some systems).

tests/data/test1034

@@ -32,8 +32,12 @@ http
 idn
 </features>
 <setenv>
-CHARSET=UTF-8
+LC_ALL=
+LC_CTYPE=en_US.UTF-8
 </setenv>
+<precheck>
+perl -MI18N::Langinfo=langinfo,CODESET -e 'die "Needs a UTF-8 locale" if (lc(langinfo(CODESET())) ne "utf-8");'
+</precheck>
  <name>
 HTTP over proxy with malformatted IDN host name
  </name>

tests/data/test1035

@@ -32,14 +32,17 @@ http
 idn
 </features>
 <setenv>
-CHARSET=UTF-8
+LC_ALL=
-LANG=en_US.UTF-8
+LC_CTYPE=en_US.UTF-8
 </setenv>
+<precheck>
+perl -MI18N::Langinfo=langinfo,CODESET -e 'die "Needs a UTF-8 locale" if (lc(langinfo(CODESET())) ne "utf-8");'
+</precheck>
  <name>
 HTTP over proxy with too long IDN host name
  </name>
  <command>
-http://too-long-IDN-name-cürl-rüles-la-la-la-dee-da-flooby-nooby.local/page/1035 -x %HOSTIP:%HTTPPORT
+http://too-long-IDN-name-cürl-rüles-la-la-la-dee-da-flooby-nooby.local/page/1035 -x %HOSTIP:%HTTPPORT
 </command>
 </client>
 
@@ -50,8 +53,8 @@ http://too-long-IDN-name-c
 ^User-Agent:.*
 </strip>
 <protocol>
-GET http://too-long-IDN-name-cürl-rüles-la-la-la-dee-da-flooby-nooby.local/page/1035 HTTP/1.1
+GET http://too-long-IDN-name-cürl-rüles-la-la-la-dee-da-flooby-nooby.local/page/1035 HTTP/1.1
-Host: too-long-IDN-name-cürl-rüles-la-la-la-dee-da-flooby-nooby.local
+Host: too-long-IDN-name-cürl-rüles-la-la-la-dee-da-flooby-nooby.local
 Accept: */*
 Proxy-Connection: Keep-Alive
 

tests/data/test165

@@ -31,9 +31,12 @@ http
 idn
 </features>
 <setenv>
-CHARSET=UTF-8
+LC_ALL=
-LANG=en_US.UTF-8
+LC_CTYPE=en_US.UTF-8
 </setenv>
+<precheck>
+perl -MI18N::Langinfo=langinfo,CODESET -e 'die "Needs a UTF-8 locale" if (lc(langinfo(CODESET())) ne "utf-8");'
+</precheck>
  <name>
 HTTP over proxy with IDN host name
  </name>

tests/data/test2046

@@ -41,9 +41,12 @@ http
 idn
 </features>
 <setenv>
-CHARSET=UTF-8
+LC_ALL=
-LANG=en_US.UTF-8
+LC_CTYPE=en_US.UTF-8
 </setenv>
+<precheck>
+perl -MI18N::Langinfo=langinfo,CODESET -e 'die "Needs a UTF-8 locale" if (lc(langinfo(CODESET())) ne "utf-8");'
+</precheck>
  <name>
 Connection re-use with IDN host name
  </name>

tests/data/test2047

@@ -42,9 +42,12 @@ http
 idn
 </features>
 <setenv>
-CHARSET=UTF-8
+LC_ALL=
-LANG=en_US.UTF-8
+LC_CTYPE=en_US.UTF-8
 </setenv>
+<precheck>
+perl -MI18N::Langinfo=langinfo,CODESET -e 'die "Needs a UTF-8 locale" if (lc(langinfo(CODESET())) ne "utf-8");'
+</precheck>
  <name>
 Connection re-use with IDN host name over HTTP proxy
  </name>