Title: 
Subject: Conference UGM
Author: -
Last saved by: elka
Company: IEEE

Usability Ranking of E-Government Website: Grey Analysis Approach   

Handaru Jati
Electronics Engineering Education Department, Yogyakarta State University, 
Karangmalang, Yogyakarta, Indonesia, 55281
handaru@uny.ac.id

Abstract—This study conducts test to measure the usability of e-government website of five Asian countries. We propose a methodology for determining and evaluating the best e-government website based on usability multicriteria. The complication of multicriteria decision making is based on the fact that those multiple criteria are often contradicting with each other, and so a solution that optimises every criterion simultaneously is generally unfeasible. Using a multicriteria decision making approach, we evaluate the performance, based on 11 usability criteria This model has been implemented by using combination of Grey Analysis (GA) and Analytical Hierarchy Process (AHP) to generate the weights for the criteria which are better and more fairly preference. Based on pairwise comparison matrices AHP transforms subjective judgements into quantified ratios of importance. Grey Analysis relates the preference of a decision maker with specially defined criterion functions.

Keywords:—Performance, e-government, quality, web diagnostic, GA and AHP model

I.	INTRODUCTION
Due to its reach, acceptance and capability to share information, the World Wide Web has become in an essential tool for institutions. Millions of websites have been created and developed and so inherently we can find every kind of website from easy to hard-to-use. Each definition of usability leads to lists of criteria about what constitutes a usability site. There are many aspect of usability, and each measure will be relevant to a particular website in varying grades. For the purpose of this research, we are going to study accessibility, navigation, site content, customization and personalization, ease of use, errors and download speed as usability criteria [1, 2]. 
First factor is accessibility, accessibility refers to the availability of a website and is a necessary factor in order to let users access the content of a page [2]. Moreover accessibility also refers to those different situations that developers should design in order to make a page accessible no matter what agent is used by users, for example language, version of a browser, different browser, among others [3]. Accessibility comprises of alternatives for multimedia presentations, readability, frames identification, and skip-navigation links (to permit users to skip repetitive navigation links) [4]. Second factor is navigation, all of the parts of the website hold together and all links inside and outside the website should work well. Broken links on the webpage also are factor that downgrades the usability of website. Each page usually has references or links or connections to other pages. These may be internal or external 

web site. Users expect each link to be valid, meaning that it leads successfully to the intended page or other resource. In the year of 2003, discovered that about one link out of every 200 disappeared each week from the Internet [5]. 
The third factor is site content. Site content concerns the accurate of the information provided. Also includes the quality of the content, amount and variety of product information [6], relevance of the content (useful) , use of media (to make content attractive), appropriate content (depth and breadth), timely /current information [7]. Number of the links, or link popularity is one of the off page factors that search engines are looking to determine the value of the webpage. Most of search engine will need a website to have at least two links pointing to their site before they will place it to their index, and the idea of this link popularity is that to increase the link popularity of a website, this website must have large amount of high quality content. Customisation and personalisation as the fourth criteria is known as made-for-the-medium [7] and characteristics of a website that fit a particular user’s needs [8]. According to these criteria, websites should provide dynamic content which has been adapted to a specific user [2]. 
The fifth factor is ease of use. Ease of use is related with the effort that is required to use it [8]. Ease of use has been seen as an important factor in determining user acceptance and behaviour in using a technology [8]. Among the subcriteria of ease of use we have the following: prioritisation of the content, structure of the website, feedback about the system status [7]. Website page optimization continues to provide significant improvements for performance and can have a large impact on its usability. The optimizations are organized into three basic categories including image, website design, and HTML code optimization. We consider the number of errors that users can make while using the webpage, how severe they are and how easy it is to recover from those errors [9] as the sixth criteria. Among sub-criteria of errors, we have number of errors, severity of the errors, and recovering from errors [9]. 
The last criterion is download speed. Suggested terms are user response time [10] or download delay [11]. This criterion is defined as the delay of instructional materials appearing on a web page after users try to access the page [11] and as a result can be affected by the content of a website [2]. The importance of this criterion is due to the fact that users become frustrated if they have to wait more than a few seconds to access all the information on a website (Nielsen (1997)). Download speed is consisted of simple and meaningful use of graphics and tables, limited use of animation, use of thumbnails [1].
II.	Literature review
III.	Methodology
Basically our research purpose have twofold aim: 1) to propose the new methodology for evaluating the quality of website and 2) to determine the best website based on the criteria proposed in the new methodology. This research examined the websites of a selected number of countries in Asia: Singapore, Korea, Japan, Hongkong, and Malaysia and covered e-government for those countries. This data of quality website will be taken more than 30 trails on various occasions on the different period of time. Using website diagnostic tools and proposed method (Analytical Hierarchy Process and Grey Relational Analysis Model) the aim of this research will be explored. All of the data for this research was taken using PC with specification: Processor Pentium Mobile 740, using Local Area Network internet connection with average bandwidth 60 kbps.  

A.	 Web Diagnostic Tools 
We used a number of widely available web diagnostic tools online, thus we used widely available website performance tool and webpage speed analyzer online service (http://www.websiteoptimization.com). List of performance measured and reported by this service include total size, number of objects (HTML, images, CSS, scripts), and download times on a 56.6 kbps connection, another available webpage online tools that we used are for testing quality is: http://validator.w3.org/checklink which was utilised in order to monitor broken links in the HTML code of the portals, while the W3C’s HTML validator website (http://validator.w3.org) was used to validate the HTML code of the portals, this standard was set up by World Wide Web Consortium (W3C), the main international standards organization for the World Wide Web. A website tool for measuring Link popularity website (www.linkpopularity.com) is used to determine the amount and quality of links that are made to a single website from many websites, this based on the page-rank analysis. 
This research also conduct using accessibility software for testing whether the webpage tested already fulfill the criteria to be accessed by people with dissabilities. This software has an ability to conduct an online test for webpage refer to the criteria setup by W3C-WCAG. Web Content Accessibility Guidelines (WCAG) is part of a series of Web accessibility guidelines published by the W3C's Web Accessibility Initiative. Accessibility software can be downloaded from www.tawdis.net. Testing using accessibility software consist of test for HTML code for knowing whether the webpage can be read by screen reader, and testing for knowing is there any alternative text for every single picture, animation, video, and audio in the webpage. 
 
B.	Sample Data
In order to get the data for this research, we examined the e-government websites from five Asian countries. These samples were not randomly selected, but a careful process was undertaken. Rather than selecting any generic websites this research attempted to evaluate the website that are considered to be leaders in the area information technology implementation based on result of a survey conducted by Waseda University for e-government website. By doing such an approach it was felt that measures of ‘best practices’ could emerge. 

C.	Grey Analysis 
Prof. Deng Julong first proposed grey system theory with the publication of his article "The Control Problems of Grey System"  [12] in an international journal in March 1982.  After more than a decade of elaboration by Prof. Deng and other domestic and foreign grey system researchers, grey system theory had grown increasingly mature, and had been applied to ten or more fields, including life science, agriculture, environmental protection, electricity, and manpower.  The grey relational analysis is a kind of method by which the related degree of every factor in the system is analyzed. The basic idea of this method is to judge the related degree by dynamic developing situation of the system. In this paper, the problem of evaluating the e-government website quality is regarded as a grey multi-objectives decision-making problem, and a grey relational evaluation model of the comprehensive situation is set up and studied practically.
	1. Setting up eigenvalue matrix
	Suppose the number of the e-government website is m, and the number of the website quality indictors is n, then a m×n eigenvalue matrix is set up.
	  （1）
	
	In formula (1), x (k) i means the value of the number i e-government website and the number k website quality indicators. Usually, two kinds of quality indicators are included, which are the maximum type indicator (the bigger the better), the minimum type indicator (the smaller the better).
	2. Standardized transformation
		Usually, it is difficult to compare between different kinds of indictors because of the different dimension. Therefore, the standardized transformation to these indictors must be done. Two formulas can be used to do this as follows:
	    						（2）
	      
								（3）
	
	Formula (2) is suitable for the maximum type indicator, Formula (3) is suitable for the minimum indicator 
	3. The calculation and analysis of the grey relational degree
	The grey relational degree can be calculated by steps as follows:
	Firstly, the absolute difference of the compared series and the referential series should get by the following  
												（4）
	and the maximum and the minimum should be find out. Secondly, choose the value of p. Here, the constant p is the distinguishing coefficient of the grey relation, and its function is to adjust the comparative environment. In practical application, we usually suppose p=0.5. Finally, calculate the relational coefficient and relational degree by formulas as follows:
	             							（5）
	
	               								（6）
	
	w(k) in Formula (5) is the proportion of the number k quality indictor to the total quality indicators. Different people can assign different weight to the same indicator, and it means that the importance of the same indictor is different to different quality assurance person. But the sum of w(k) is 100%. The result by using Formula (5) can be used to measure the comprehensive quality situation of the e-government website.

D.	Analytical Hierarchy Process
	This section should focus on the actual achievement for the semester. In-depth discussions and analysis are expected from the on-going work while summary and critical analysis of other related works may appear for first report. Analytic Hierarchy Process (AHP) was originally designed by Saaty [13] to solve complicated multi-criteria decision problem, beside that AHP is appropriate whenever a target is obviously declared and a set of relevant criteria and alternatives are offered [14]. AHP has been proposed for determining the best website to support researcher through the decision making activity, which aims to determine the best website among pool of e-government website. AHP is a popular model to aggregate multiple criteria for decision making [15]. In AHP the problems are usually presented in a hierarchical structure and the decision maker is guided throughout a subsequent series of pairwise comparisons to express the relative strength of the elements in the hierarchy. In general the hierarchy structure encompasses of three levels, where the top level represents the goal, and the lowest level has the website under consideration. The intermediate level contains the criteria under which each website is evaluated. The final score obtain for each website across each criterion is calculated by multiplying the weight of each criterion with the weight of each website. Website which has got the highest score is suggested as the best website and decision maker may consider that one as the best decision choice.
	Generally, AHP has the following steps:
1.	Employ a pair-wise comparison approach. Fundamental scale for pair-wise comparisons developed to solve this problem [13]. The pair-wise comparison matrix A, in which the element of the matrix is the relative importance of the  factor with respect to the  factor, could be calculated 
as                    (7)
2.	There are /judgments required for developing the set of matrices in step 1. Reciprocals are automatically assigned to each pair-wise comparison, where  is the matrix size.

Table2. Each of membership functions’ parameter
| AHP
Equal | 1
Equal –Moderate | 2
Moderate | 3
Moderate- Fairly Strong | 4
Fairly Strong | 5
Fairly Strong- Very Strong | 6
Very Strong | 7
Very Strong- Absolute | 8
Absolute | 9

IV.	Result and Discussion 
In order to provide format consistency, the following guidelines are to be followed. Results of the e-goverment websites test based on load time, response time, page rank, frequency of update, traffic, design optimization, size, number of items, accessibility error, markup validation, and broken link are showed in table 3. 

Table3. Testing Result for Websites Performance Based on Criteria
quality Criteria | .sg | .kor | .jp | .hk | .my
load time | 30.77 | 0.30 | 68.93 | 41.94 | 77.51
response time | 1.94 | 1.17 | 1.73 | 1.03 | 4.84
page rank | 2870.00 | 430.00 | 2020.00 | 9690.00 | 2470.00
frequency of update | 60.00 | 60.00 | 60.00 | 1.00 | 60.00
Traffic | 62000.00 | 39800.00 | 223200.00 | 499600.00 | 228200.00
design optimization | 37.50 | 57.00 | 36.50 | 33.00 | 22.00
Size | 128305.00 | 511.00 | 285645.00 | 195384.00 | 366825.00
Number of items | 26.00 | 1.00 | 60.00 | 15.00 | 22.00
accessibility error | 37.00 | 0.00 | 2.00 | 0.00 | 15.00
markup validation | 79.00 | 5.00 | 21.00 | 3.00 | 80.00
broken link | 4.00 | 0.00 | 1.00 | 1.00 | 9.00

Korea 

Hongkong

Malaysia 

Japan 

Singapore 

Selecting Best E-Government Website

Load time

Response time

Page rank

Frequency of update

traffic

Design 

size

Number of items

Accessibility error

Markup validation

Broken link

Figure 1 FAHP/AHP Model of Best Websites

The last step in this method is to compute the final score of each website. Then get the sum of each column and the sum represents the score of each single website.
In order to provide format consistency, the following guidelines are to be followed. Results of the e-goverment websites test based on load time, response time, page rank, frequency of update, traffic, design optimization, size, number of items, accessibility error, markup validation, and broken link are showed in table 3. 

Table3. Testing Result for Websites Performance Based on Criteria
quality Criteria | .sg | .kor | .jp | .hk | .my
load time | 30.77 | 0.30 | 68.93 | 41.94 | 77.51
response time | 1.94 | 1.17 | 1.73 | 1.03 | 4.84
page rank | 2870.00 | 430.00 | 2020.00 | 9690.00 | 2470.00
frequency of update | 60.00 | 60.00 | 60.00 | 1.00 | 60.00
Traffic | 62000.00 | 39800.00 | 223200.00 | 499600.00 | 228200.00
design optimization | 37.50 | 57.00 | 36.50 | 33.00 | 22.00
Size | 128305.00 | 511.00 | 285645.00 | 195384.00 | 366825.00
Number of items | 26.00 | 1.00 | 60.00 | 15.00 | 22.00
accessibility error | 37.00 | 0.00 | 2.00 | 0.00 | 15.00
markup validation | 79.00 | 5.00 | 21.00 | 3.00 | 80.00
broken link | 4.00 | 0.00 | 1.00 | 1.00 | 9.00

Korea 

Hongkong

Malaysia 

Japan 

Singapore 

Selecting Best E-Government Website

Load time

Response time

Page rank

Frequency of update

traffic

Design 

size

Number of items

Accessibility error

Markup validation

Broken link

Figure 1 FAHP/AHP Model of Best Websites

The last step in this method is to compute the final score of each website. Then get the sum of each column and the sum represents the score of each single website.

Table 6. The compared series and the referential series
| A | B | C | D | E | F | G | H | I | J | K
X0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1
X1(.sg) | 0.61 | 0.76 | 0.26 | 1.00 | 0.05 | 0.44 | 0.65 | 0.58 | 0.00 | 0.01 | 0.56
X2(.kor) | 1.00 | 0.96 | 0.00 | 1.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.97 | 1.00
X3(.jp) | 0.11 | 0.81 | 0.17 | 1.00 | 0.40 | 0.41 | 0.22 | 0.00 | 0.95 | 0.77 | 0.89
X4(.hk) | 0.46 | 1.00 | 1.00 | 0.00 | 1.00 | 0.31 | 0.47 | 0.76 | 1.00 | 1.00 | 0.89
X5(.my) | 0.00 | 0.00 | 0.22 | 1.00 | 0.41 | 0.00 | 0.00 | 0.64 | 0.59 | 0.00 | 0.00

Calculate the absolute difference of the compared series and the referential series by formula (4) and find out the 

maximum and the minimum and the results displayed in Table 7. 

Table 7. The absolute difference of the compared series and the referential series
Δx1 | 0.39 | 0.24 | 0.74 | 0.00 | 0.95 | 0.56 | 0.35 | 0.42 | 1.00 | 0.99 | 0.44
Δx2 | 0.00 | 0.04 | 1.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.03 | 0.00
Δx3 | 0.89 | 0.19 | 0.83 | 0.00 | 0.60 | 0.59 | 0.78 | 1.00 | 0.05 | 0.23 | 0.11
Δx4 | 0.54 | 0.00 | 0.00 | 1.00 | 0.00 | 0.69 | 0.53 | 0.24 | 0.00 | 0.00 | 0.11
Δx5 | 1.00 | 1.00 | 0.78 | 0.00 | 0.59 | 1.00 | 1.00 | 0.36 | 0.41 | 1.00 | 1.00

		Then suppose we set p = 0.5. And the grey relational coefficient can get by Formula (5) and displayed in Table 8. 

Table 8. Grey Relational Coefficient 
§1(.sg) | 0.559 | 0.675 | 0.404 | 1.000 | 0.344 | 0.473 | 0.589 | 0.541 | 0.333 | 0.336 | 0.529
§2(.kor) | 1.000 | 0.929 | 0.333 | 1.000 | 0.333 | 1.000 | 1.000 | 1.000 | 1.000 | 0.951 | 1.000
§3(.jp) | 0.360 | 0.730 | 0.376 | 1.000 | 0.454 | 0.461 | 0.391 | 0.333 | 0.902 | 0.681 | 0.818
§4(.hk) | 0.481 | 1.000 | 1.000 | 0.333 | 1.000 | 0.422 | 0.485 | 0.678 | 1.000 | 1.000 | 0.818
§5(.my) | 0.333 | 0.333 | 0.391 | 1.000 | 0.459 | 0.333 | 0.333 | 0.584 | 0.552 | 0.333 | 0.333
	
	In Formula (6), w(k) to every indictor and to every quality assurance may be different. According to the AHP judgment, we can suppose: w(A)=0.270, w(B)=0.197, w(C)=0.148, (D)=0.107, w(E)=0.076, w(6)=0.052, w(7)=0.042, w(8)=0.042, w(9)=0.030, w(10)=0.021, and w(11)= 0.016. Then the relational degree can easily get by using Formula (6). The final score obtained for each website across each criterion was calculated by multiplying the weight of each criterion with the weight of each website. Website which has the highest score is suggested as the best website and decision maker may consider that one as the best decision choice. Table 9 depicts the weight scores of websites using combination of Grey Analysis and AHP model evaluation. The most important thing is regarding the 
final results, the website which has the highest score is suggested as the best website for the proposed GA and AHP model.
GA and AHP method combine evaluation method used before. This model has to assign weights to the criteria that involve in decision making process. Weight for alternative is taken from GA process and weight for criteria is taken from AHP. By applying evaluation model between GA and AHP approach for website evaluation has resulted in significant reducing of computation, raised the overall speed and effectiveness with respect to the underlying methodology and ultimately enabled more efficient and significantly procedure compared with other methods. 

Table 9. Final Result for e-government website (GA and AHP Model)
| | .sg | .kor | .jp | .hk | .my
A | Max | 0.151 | 0.270 | 0.097 | 0.130 | 0.090
B | Max | 0.133 | 0.183 | 0.143 | 0.197 | 0.066
C | Min | 0.060 | 0.049 | 0.056 | 0.148 | 0.058
D | Min | 0.107 | 0.107 | 0.107 | 0.036 | 0.107
E | Min | 0.026 | 0.025 | 0.034 | 0.076 | 0.035
F | Min | 0.025 | 0.052 | 0.024 | 0.022 | 0.017
G | Max | 0.024 | 0.042 | 0.016 | 0.020 | 0.014
H | Max | 0.022 | 0.042 | 0.014 | 0.028 | 0.024
I | Max | 0.010 | 0.030 | 0.027 | 0.030 | 0.017
J | Max | 0.007 | 0.020 | 0.015 | 0.021 | 0.007
K | Max | 0.008 | 0.016 | 0.013 | 0.013 | 0.005
Sum | | 0.574 | 0.836 | 0.547 | 0.721 | 0.440
Rank | | 3 | 1 | 4 | 2 | 5

Table 9 depicts the final scores of websites. The most important thing is regarding the final results, the website which has the highest score is suggested as the best website for the proposed GA and AHP model. In accordance with the results generated by the proposed model, Korea e-government website has the highest score of 0.836 in comparison with the rest of e-government websites. As a result, the proposed GA and AHP model rank for e-government website is: Korea (score: 0.836), Hongkong (score: 0.721), Singapore (score: 0.574), Japan (score: 0.547), and the last rank are Malaysia (score: 0.440).  
I.	Conclusion 
In this paper we evaluate the quality of Asian e-government websites. Using a series of online diagnostic tolls, we examined many dimensions of quality, and each dimension was measured by a specific test online. The result of this study confirmed that the website presence of Asian e-government website is neglecting performance and quality criteria. It is clear in our research that more effort is required to meet these criteria in the context of website design. This suggests that web developers responsible for e-government website should follow and encourage the use of recognised guidelines when designing website. To get results on the quality of a website, we measured sample data from e-government website in 5 Asian countries and calculate load time, response time, page rank, frequency of update, traffic, design optimization, page size, number of item, accessibility error, markup validation, and broken link. We proposed methodology for determining and evaluating the best e-government sites based on many criteria of website quality based on Grey Analysis and Analytical Hierarchy Process. This new combination model has been implemented to generate the weights for the criteria which are much better and guaranteed more fairly preference of criteria. Applying combination model between GA and AHP approach for website evaluation has resulted in significant acceleration of implementation, raised the overall effectiveness with respect to the underlying methodology and ultimately enabled more efficient and significant. Korea website has the highest in comparison with the rest of e-government websites. As a result, in overall ranking for the proposed model rank for e-government website is: Korea, Hongkong, Singapore, Japan, and Malaysia. Limitation of this research occurred in the number of sample size and time factor. This research used limited sample size of 30 data and taken during a short period of observation time. Future directions for this research are added criteria for evaluating websites quality, such as availability and security aspect, also from the cultural perspective, since culture has an impact upon a website. Another approach also can be conducted for other service sectors such as e-business and academic website. Moreover because the ultimate determinant of quality website is the users, future directions for this research also involve the objective and subjective views of the e-government website from user’s perspective. 

References

[1]	1	Turban, E., and Gehrke, D.: ‘Determinants of e-commerce website’, Human Systems Management, 2000, 19, (2), pp. 111-120
[2]	2	Pearson, J.M., and Pearson, A.M.: ‘An exploratory study into determining the relative importance of key criteria in web usability: a multi-criteria approach’, Journal of Computer Information Systems, 2008, 48, (4), pp. 115-127
[3]	3	World Wide Web, C.: ‘Web content accessibility guidelines 1.0’, Retrieved April, 1999, 3, pp. 2005
[4]	4	http://webaccess.tamu.edu/, accessed 25 January 2012
[5]	5	McCowen, F., Michael, N., and Bollen, J.: ‘The Availability and Persistence of Web References in D-Lib Magazine’. Proc. the 5th International Web Archiving Workshop and Digital Preservation (IWAW'05), Viena 2005 pp. Pages
[6]	6	Palmer, J.W.: ‘Web Site Usability, Design, and’, Information systems research, 2002, 13, (2)
[7]	7	Keeker, K.: ‘Improving web site usability and appeal’, Retrieved May, 1997, 19, pp. 2002
[8]	8	Agarwal, R., and Venkatesh, V.: ‘Assessing a Firmâ€™s Web Presence: A’, Information systems research, 2002, 13, (2)
[9]	9	Nielsen, J.: ‘Designing web usability’ (Pearson Education, 2004. 2004)
[10]	10	Nielsen, J.: ‘Why you only need to test with 5 users’, Test, 2000, 19, (September 23)
[11]	11	Davis, E.S., and Hantula, D.A.: ‘The effects of download delay on performance and end-user satisfaction in an Internet tutorial’, Computers in Human Behavior, 2001, 17, (3), pp. 249-268
[12]	12	Ju-Long, D.: ‘Control problems of grey systems’, Systems & Control Letters, 1982, 1, (5), pp. 288-294
[13]	13	Saaty, T.: ‘The analytic hierarchy process: planning, priority setting, resources allocation’ (McGraw-Hill, 1980. 1980)
[14]	14	Ozden, B., and Karpak, B.: ‘An AHP application in vendor selection’, in Editor (Ed.)^(Eds.): ‘Book An AHP application in vendor selection’ (ISAHP, 2005, edn.), pp. 
[15]	15	Yuen, K., and Lau, H.: ‘Software vendor selection using fuzzy analytic hierarchy process with ISO/IEC9126’, IAENG International journal of computer science, 2008, 35, (3), pp. 267-274.
[16]	
[17]