Transcription

Journal of Theoretical and Applied Information Technology28th February 2019. Vol.97. No 4 2005 – ongoing JATIT & LLSISSN: 1992-8645www.jatit.orgE-ISSN: 1817-3195USER EXPERIENCE QUESTIONER AND HEURISTICSEVALUATION IN ONLINE LEARNING ENVIRONMENT1SITI NUR AISYIAH SYAHRIR, 2SFENRIANTO12Information Systems Management Department, BINUS Graduate Program – Master of InformationSystems Management, Bina Nusantara University, Jakarta, Indonesia 11480E-mail: [email protected], [email protected] learning has an important role to support learning activities. Although it has been widely used byvarious universities, there is always effort to increase the interest of the user using a system of onlinelearning which can be measured through user experience. Several quantitative and qualitative studies havebeen conducted to measure online learning from the aspect of the user experience; however, such a study isstill arranged separately. This research discusses user experience measurement using two methods, namelyuser experience questioner (UEQ) as a quantitative study and evaluation heuristics (HE) as a qualitativestudy. The data retrieving in the measurement of user experience is run by distributing questionnaires to100 students of the online-learning program in University xyz. Subsequently, the data was analyzed byemploying UEQ and HE. The research results showed that the input and suggestions form users onlinelearning is relatively similar. Results on UEQ attractiveness resulted in highest value and the novelty aspectof the lowest. While the results HE all aspects which had a good evaluation of the results, however there isstill deficiencies which in the measurement HE will further detailing of which parts should be more focusedon. On the final results, there are suggestions that HE may be the fix for combining UEQ and HE cannot bedone, because each has its own distinctive aspects in the measurement. But both can complement theresults of research with their respective advantages.Keywords: UX; User experience; the questioner heuristics evaluation; Online learning1.INTRODUCTIONNow, various universities in Indonesiahave been utilizing the online learning to supportteaching and learning. University xyz is one of theprivate universities in Indonesia that haveimplemented the online learning (xyz OnlineLearning) in teaching and learning activities for allstudents: in various regions and foreign countries.xyz Online Learning is e-Learning System for thenecessities of teaching and learning activitiesranging from email, schedule, grades, discussionforums, students and professors, assignmentuploads, sharing information and bill paymentinformation.However, although xyz Online Learninghas been implemented for approximately sevenyears, the system has not been tested for userexperience, so it does not explicate how wellstudents use the system and what improvementsshould be made. As a system, xyz Online Learningcertainly has advantages, and the shortcomings ofthe students as users. The interest of users using asystem can be distinguished by measuring the userexperience viewed from the perspective of the user[1]. Measurement of user experience aims toevaluate a system in which the results of themeasurement of each aspect of the user experiencecan produce a recommendation in its development[2].Some studies have measured the userexperience of an e-Learning system, such as:measuring e-Learning based user experienceQuestioner (UEQ) [3][4] and using a heuristicevaluation (HE) in the online learning environment[5][6]. The UEQ approach is a quantitative methodof quantitative user experience with the analyzedmodel of the 6 aspects of Attractiveness,Efficiency, Perspicuity, Dependability, Stimulation,and Novelty [7].While the HE approach, qualitativemeasurements that pay attention to 10 aspects:visibility of system status, match between systemand the real world, user control and freedom,1071

Journal of Theoretical and Applied Information Technology28th February 2019. Vol.97. No 4 2005 – ongoing JATIT & LLSISSN: 1992-8645www.jatit.orgconsistency and standards, error prevention,recognition rather than recall, flexibility andefficiency of use, aesthetic and minimalist design,help users recognize, diagnose and recover fromerrors, help, and documentation [8] Nevertheless,existing research to measure user experience onlinelearning has not been done thoroughly in usingUEQ f and HE.Therefore, to estimate the user experienceas a whole and complement, each other can be usedboth methods. This research is sharpened toovercome the problems in apprehending the userexperience of students at xyz Online Learning inquantitative study using UEQ and qualitative studyby using HE.E-ISSN: 1817-3195methods, such as User Experience Question (UEQ)Heuristics Evaluation (HE), and so on. UEQ is atool for measuring the user experience of a systemand is available on the website http://www.ueqonline.org/ The UEQ questionnaire is quantitativeand is available in various languages, such asEnglish, German, Spanish, Chinese, Indonesian,and so on. Figure 1 shows the UEQ Englishversion.2. RESEARCH METHODOLOGY2.1. Data setsData collection was performed bydistributing UEQ and HE questionnaires online byusing Google form facility to the online students ofUniversity xyz. There were 100 students who fillthe UEQ and HE questionnaires.Table 1. Demographic analysis for participantsVariableParticipantSocial media usedDataOnline learning student :Undergraduate (50 participant)Graduate (50 participant)Google formWhatsup (with link to connectgoogle form)DM Instagram (with link toconnect google form)Figure 1. UEQ English Version [7]The form of the UEQ questionnaire hasbeen described in figure 1 above. While the HEquestionnaire was done by considering the 10aspects associated with online learning system ofUniversity xyz. Subsequently, provided a columnthat people can fill in according to their experiencesand suggestions.2.2. Related worksUser experience discusses how users feeland what they feel when using a product/system [9]The purpose of User Experience is to know whetherthe user is comfortable, the system design is simple,easy to accept by the user, the use of the processcan give comfort and access as per the user[10]Measurement of user experience using severalFigure 1 shows 26 questions with a 7likert scale, with intervals of 1 to 7 [11] The UEQquestion represents six aspects of user experiencemeasurement: which are (1) Attractiveness as towhether the system/product is favored/disliked byits users; (2) Efficiency related to whether thesystem/product is easy to use and easy to learn; (3)Perspicuity related to whether the user resolves theproblem on the system without effort assistance; (4)Dependability as to whether the user perceives theinteraction in a controlled system; (5) Stimulationof whether the system/product makes the usermotivated to use it; and (6) Novelty whether theproduct is innovative, creative and appealing [12][13] [14].1072

Journal of Theoretical and Applied Information Technology28th February 2019. Vol.97. No 4 2005 – ongoing JATIT & LLSISSN: 1992-8645www.jatit.orgTo calculate the interval of UEQ is from -3(horribly bad) to 3 (extremely good) [7]. Figure 2is an example of a graph results UEQ for resultsexcellent. Tabel 1 explain, 26 questions will be.E-ISSN: 1817-3195grouped based on 6 UEQ aspects which will bedistributed with the number of questionnairesparticipants so that it gets the average score resulton every aspect of UEQ.Table 2. Grouping questions based on aspects of UEQAspects of yStimulationNoveltyNo. Questions1. annoying/enjoyable; 12. good/bad; 14. unlikable/pleasing; 16. Theunpleasant/pleasant; 24.attractive / unattractive; 25.friendly / unfriendly2. not understandable / understandable; 4. easy to learn / difficult to learn;13. complicated / easy; 21. clear / confusing9. fast / slow; 20.inefficient / efficient; 22. impractical / practical; 23.organized / cluttered8. unpredictable / predictable; 11. obstructive / supportive; 17. secure/insecure; 19. meet the expectations / does not meet the expectations5. valuable / inferior; 6. boring / exciting; 7. not interesting / interesting; 18.motivating / demotivating3. creative / dull; 10. inventive / conventional; 15. usual / leading edge; 26.conservative / innovativeFigure 2. Samples of products with Good Results [7]User Experience measurements usingUEQ have been used to measure e-learning at theFaculty of Computer Science, the University ofIndonesia[3]. Then the measurement of userexperience using UEQ against iDigital Museumsystem, with a case study of interactive applicationsfor the theme of Bandung Purba [13]. HE methodis an evaluation of user experience on a systembased on qualitative feedback from users. Table 1shows the 10 aspects measured using a HE [14].Table 3. Aspect HE and the purposesAspectsVisibility of systemstatus (A1)Match between systemand the real world (A2)User control and freedom(A3)Consistency andstandards (A4)PurposesThe system should always provide information about what process is goingthrough a feedback.The system should speak in accordance with the user's language.Users often make mistakes in the selection of system functions and want toimmediately leave, all it takes is an undo.User should not have a different idea of a word, situation or action, all followingan existing platform, by agreement or by default,1073

Journal of Theoretical and Applied Information Technology28th February 2019. Vol.97. No 4 2005 – ongoing JATIT & LLSISSN: 1992-8645www.jatit.orgError-prevention (A5)Recognition rather thanrecall (A6)Flexibility and efficiencyof the usage of (A7)Help users recognizediagnose and recoverfrom errors (A8)Aesthetic and minimalistdesign (A9)Help and documentation(A10)E-ISSN: 1817-3195A good system will be designed to prevent users from making mistakes, bydisplaying a confirmation selection of the activities the user will performMinimize the users in thinking or remembering things against objects, actions,and choices. All instructions in the system are clear and easy for the user.Acceleration, meaning the system is made in order to work fast and easy to useeven though the user is new or expertError messages should be described using clear language (not using code),explaining the problem, and providing suggestions as solutionsThe dialog that appears should not contain irrelevant or unnecessary informationThe need for help and documentation of system usage.The HE method has been used in theonline shopping portal evaluation at LublinUniversity of Technology, Poland. The resultshows that the testing of expert user experience ismore focused on the interface system [5]. Table 3shows the HE questionnaire to measure userexperience to online students of University xyz.Table 4. The design of the HE Questionnaire is based on Online Learning xyzAspectsA1A2A3System Online Learning xyzDisplay and icon that are displayed are familiar, easy-to-understand,attractive and information feedback is available as it could giveconfidence that the process has been conducted.Use English consistently on every feature and menu in the system(A4)The use of the feature back on the system in accordance with the behaviorof usersStandardization on the display toolbar and icon in the system(A5)system well provides the confirmation regarding activities or choices(A6)The icons that are used in the system are familiar for its users(A7)(A9)The system is easily understood by its users and undergoes infrequentproblemsthe system provides an error message with a language that is easy to beunderstood by usersWarning on dashboard system is in compliance with the needs of users(A10)Warning on dashboard system is in compliance with the needs of users(A8)Experiences andSuggestions userExperiences: .Suggestions: .Experiences: .Suggestions: .Experiences: .Suggestions: .Experiences: .Suggestions: .Experiences: .Suggestions: .Experiences: .Suggestions: .Experiences: .Suggestions: .Experiences: .Suggestions: .Experiences: .Suggestions: .Experiences: .Suggestions: .4. RESULTIn this section described the results of themeasurements UEQ and HE. This result as thebasis for development of recommendations xyzonline learning.4.1 Measurement of UEQFigure 3 and Table 4 show the results ofthe UEQ measurement of online learning at1074

Journal of Theoretical and Applied Information Technology28th February 2019. Vol.97. No 4 2005 – ongoing JATIT & LLSISSN: 1992-8645www.jatit.orgUniversity xyz using 6 aspects. If aspect is includedin the interval of 0.0 up to 3.0 points, so it can beconcluded that the result is neutral (standard), and ifinterval of -0.3 up to 0.0 points, so it can beconcluded that the result is not standard.Attractiveness was at 1,118 points. It can beconcluded that the result is neutral (standard).These results indicated that xyz online learning isalready good enough in some existing measurementin aspect of attractiveness that is: according to thestudent, the system is quite good, pleasant,pleasing, friendly, attractive and pleasant.Then, perspicuity was at points 1,015. It itcan be concluded that the result is neutral(standard). These results indicate that xyz onlinelearning is already good enough in some existingmeasurement in aspect of perspicuity that isaccording to the student system is quiteunderstandable, easy to learn, easy (simple) andclear. Efficiency was at the point of 0.910. It can beconcluded that the result is neutral (standard).These results indicated that xyz online learning isalready good enough in some existing measurement3210‐1‐2‐31.1181.015E-ISSN: 1817-3195in aspect of efficiency: according to the student,system is quite efficient, fast, well organized andpractical.Dependability was at points of 1,043. Itcan be concluded the result is neutral (standard).These results indicated that xyz online learning isalready good enough in the aspect of dependability,that according to students, the system is supportingenough, predictable, secure and fulfilling theexpectations. Stimulation is on points of 0.983. Itcan be concluded that the result is neutral(standard). These results indicated that xyz onlinelearning is already good enough in some existingmeasurement in aspect of stimulation: according tothe student system is quite useful, engrossing(existing), interesting and motivating.While, novelty is at the points of 0.333. Itcan be concluded the result is neutral (standard).These results indicated that xyz online learning isalready good enough in some existing measurementin aspect of efficiency: according to the student,system is quite efficient, fast, well organized andpractical.0.9101.0430.9830.333Figure 3. Is a Graph results of measurement aspect UEQTable 5. Scale measurement resultsBased on the results of the measurements,aspect of attractiveness, online learning platform ofuniversity xyz got the highest points while thelowest points is novelty aspect; the overall pointsare indeed on the evaluation standard of points orintervals is quite good, however by the presence4.2 Measurement of HEone of the aspects that have a low value in aspect ofnovelty, it held a further evaluation of some pointson the novelty aspect i.e. creative, inventive,leading and innovative systems to improve qualityxyz online learning.1075

Journal of Theoretical and Applied Information Technology28th February 2019. Vol.97. No 4 2005 – ongoing JATIT & LLSISSN: 1992-8645www.jatit.orgE-ISSN: 1817-3195from HE at xyz online learning online learningusing 10 aspects. A table 4-13 below shows theresult of HE for each aspects [8].Measurement results from HE at xyzonline learning using 10 aspects. Table 5 shows theresult of HE for each aspect. Measurement resultsTable 6. Results HE aspects of visibility of system statusAspect(A1)CommentOpinionAdvice/expected on systems Display and icon that are Give information on icon labelsthat are not too familiar to users.displayed are familiar but thereare some icons are not familiar If you have to write in the forum,and less understandable, as forshould not appear reminder emailexample in the section existagain.schedule feature icon on the chat Suggestions for system now ificon, instructions on the systemclick the forum on schedule thenare obvious.display the dedicated forum less Thesystemprettyeasilyfriendly, uncomfortable in such aunderstood. looks interestingmess, look different if we directlyclick the forums will appear the Every open dashboard noticeschedules arranged according toalways appear a little annoying ifmess with the date.you're hurry want to access tasks.In general, students argued that aspects ofthe visibility of system status on the online learningxyz are familiar and easy to understand. However,on the dashboard needs to be fixed. Therefore, theadvice given students to note such as addinginformation on icon labels are not familiar and mostopinion that.Table 7. Results HE Aspects match between system and the real worldAspect(A2)CommentOpinionAdvice/expected on systems Thelanguageusedis Better using good languagemenu type 1 as well as theunderstandable and tailored to itscontent, but the user can replaceusers.it with another language if they The use of the language waswish.already consistent for every feature There has not been a selectionof the menu that is using Englishof the language.In general, students argued that aspectsmatch the system and the real world in onlinelearning xyz learning as it has used language that iseasy to understand and consistent in every menu.However, some features are not yet available inseveral languages. Therefore, the advice givenstudents to note such as add a language selection onthe system in accordance with the language of theuser.1076

Journal of Theoretical and Applied Information Technology28th February 2019. Vol.97. No 4 2005 – ongoing JATIT & LLSISSN: 1992-8645www.jatit.orgE-ISSN: 1817-3195Table 8. Results HE aspects of user control and freedomAspect(A3) CommentOpinionAdvice/expected on systemsThe user controls are indeed needed It should provide a sitemap orguidance in its use in order tobecause users often use the featuredmake it easier for its users.back on smartphone.Use of the back button depending Evaluation suggestions menuback by viewing habits of users,on user behavior.so that users are comfortableOn the web, there is no direct backusing the systemfunction, its weaknesses and weshould wait for a few seconds in Advice we recommend some ofthe menu can be openedorder to return to the previoussimultaneously in the new tab.display menu and mostly alsocannot open in multiple tabs at thesame time, the example assignmentSystem has a function that is not ina scope or main menus not alwaysappearing. Users often are forced tofollow the flow of the systemwithout free choice.In general students argued that aspects ofuser control and freedom in online learning xyz interms of user controls assessed student use of thefunction of the back depending on the behavior ofits users. However, back on the menu in the systemneed to be further evaluated by looking at the user'shabits. Therefore, the advice given by students hasto be noted such as create a menu on the system canbe opened in a new tab the information label on theicon which is not familiar and most opinionssuggest that the system need to provide usageinstructions.Table 9. Results consistency Aspect of HE and standardsAspect( A4)CommentOpinionAdvice/expected on systems Display, icon, toolbar and others in consistency in the display of themenu and the very Languagethe system are already good andneeded in the system thereforestandardshould look familiar and the The menu is displayed for a certainlanguage used should be cleardifferent options with other options,also avoid ambiguous wordsso that the frame of the built systemfor users is quite confusing becauseit does not have the same base.In general students argued that aspects ofconsistency and standards on online learningsystem overall overview online learning xyz isalready good and consistent. However, there aresome menu when selected by the user is stillmaking confused so that needs to be fixed.Therefore, the advice given students to note such aspaying attention to the display's menu and languageon the system.1077

Journal of Theoretical and Applied Information Technology28th February 2019. Vol.97. No 4 2005 – ongoing JATIT & LLSISSN: 1992-8645www.jatit.orgE-ISSN: 1817-3195Table 10. Result of HE Aspect error-preventionAspect( A5)CommentOpinionAdvice/expected on systems On some systems there are already We recommend that you addedcaptions (kind of tools tips)activities against confirmation/sentences when the cursor isvotes conducted so that enoughpointed to one of the menu.makes it easy Suggestions on the Forum onmobile applications can post ananswer blank when notdepressed Reply buttonIn general students argued aspects of theerror-prevention on online learning has beenproviding confirmation that online learning xyz insome action in the system have been good.However, on the use of the via mobile, there is stillthe problem of IE if accidentally in reply then itwould empty answer will be posted without noticeof these things need to be fixed. Therefore, theadvice given students to note such as addingannotations (box informers) if the cursor is broughtnear in the menu.Table 11. Results HE aspects of recognition rather than recallAspect( A6)CommentOpinionAdvice/expected on systems Icons chosen has been To review menu chat among students inthe system, probably it is morevery familiar so that itcomfortable if student can connect to othermakes ease for users tomediasuse In private Chat menu onthe system is somewhatrarely used because usersare more y.In general students argued aspects ofrecognition rather than recall on online learning xyzselection icons are familiar. However, on the chatmenu for intercultural students are very rarely used.Therefore, the advice given students to note such asre-evaluating the usability features of chat forstudents to look at their convenience when usingthe system.Table 12. Results HE aspects of flexibility and efficiency of the usageAspect(A7)CommentOpinionAdvice/expected on systems The system is easy to Advice: should be more facilitated onchoice of menus for example if one is tounderstand but lessuse the design of Microsoft because mostpractical and sometimespeople are familiar with the designconfusingfornewpeople. Suggestion on the part of the functionclick on each module of subject that Cannot right click openusually appears on the dashboard, itin new tab.cannot open new tab. If we use the Often going down by afunction open new tab then that tab isparticular system onsimply a blank will appear It feels like thissome times1078

Journal of Theoretical and Applied Information Technology28th February 2019. Vol.97. No 4 2005 – ongoing JATIT & LLSISSN: 1992-8645www.jatit.orgE-ISSN: 1817-3195should be remedied because every periodnot only contains 1 courses, but somesubjects are very stiff if the user shouldopen the procedurally per each subject.So, it should be multi tab in it. System performance must be at themonitor at regular intervals to avoid downtime at certain times.In general students argued that aspects ofthe flexibility and efficiency of the usage of ononline learning xyz systems have easy tounderstand. However, it often happens down thesystem. The menu system is also not able to clickon the new tab to add them to these needs to befixed. Therefore, the advice given students to note,such as the need to perform system monitoring on aregular basis and make the system able to perform anew tab on each menu.Table 13. Results of HE Aspects help users recognize diagnose and recover from errorsAspect(A8)CommentOpinionAdvice/expected on systems Suggestionerrormessage The error message was clear.should be detailed but not His"success"textwhenexcessive and by using clearuploading was error sometimes.language so that users will beSometimes when the failedmore quickly know his faultsupload task is not accompaniedby an error message, just swirl it,While if there is writing directsuccess successful but not directupload tab closedIn general, students argued that Aspectshelp users recognize diagnose and recover fromerrors on online learning xyz message was clear.However, the info when the action successfully onthe system error sometimes this needs to be fixed.Therefore, the advice given the student a morenoteworthy error messages so that users readilyunderstand.Table 14. Results HE Aspects of aesthetic and minimalist designAspect(A9)CommentOpinionAdvice/expected on systems Design a bulletin board on the Suggestions the announcementshould be made in the form ofsystem's been good but Warningslides, so when openingdashboard is irrelevant. studentsdashboard doesn't show upmust read the news over and overevery time.again for news not for the student(suchasgraduation Might be better differentiatedannouncements etc.) informationinformation dialog again if onthat appears should not beeach host. Short and clearcombined with students atdialogue is needed in thedifferentlevelssothatsysteminformation is right on target1079

Journal of Theoretical and Applied Information Technology28th February 2019. Vol.97. No 4 2005 – ongoing JATIT & LLSISSN: 1992-8645www.jatit.orgIn general students argued aspects of theaesthetic and minimalist design on online learningxyz on menu boards is already good. However,votes are fewer contents less relevant it needs to beE-ISSN: 1817-3195fixed. Therefore, the advice given students to notethat is to select the contents of the notice boards inaccordance force of each student (information is notuniversal).Table 15. Results HE aspects of help and ted on systems the Help menu provided such as Indispensable Advice: manualbook for students for the use ofthe for ask question (FAQ) on aLMS because short tutorial givensystem is already well, butat the time of introduction of thehaven't quite helpful whensystem of learning activities onstudents are having problemsline is not sufficient to provideusing the systemthe knowledge for new students Usually for new students is stillin recognizing the features thatvery difficult to learn so needare present in the LMS. Betterhelp menu.existence tutorial using the Yet the existence of thesystem is in the form of a video.documentation of the use of the Supplementarymenuchatsystem can be kept by themessageinthesystemtogetthestudent while accessing thehelp of Admins.website. The need for the feature to uploadproof to capture if the system isexperiencing an interferenceIn general students argued aspects of thehelp and documentation on online learning xyz likemenu for ask question (FAQ) provided is alreadygood. However, it is reasonably less help studentsneeds to be fixed. Therefore, the advice givenstudents to note such as adding a menu chat directly5. CONCLUSIONUser experience measurement resultsusing UEQ showed study quantitative andqualitative studies UEQ measurement at universityxyz, online learning shows all the aspects of theresult is positive, despite the novelty aspect is verymuch the point between the points obtained by theother aspects namely 0.8. Online learning xyz isexpected to improve this aspect of novelty. HEmeasurement results according to a studentdemonstrating all aspects have been good and needdevelopment. However, in every aspect there arestill drawbacks. Therefore, all student suggestionsfor development and improvement need to beaccommodated by the online learning xyz.Measuring user experiences complement eachother, using the method of quantitativemeasurements as UEQ and methods HE asqualitative measurement.with the admin when having problems while usingthe video and also provide another alternative tointroduce the use of online learning at universityxyz for example student manual book use or can bein the form.When conducting research using twomethods, the thing that must be considered was thereadiness of respondents to provide theirwillingness to fill out questionnaires. Because therespondents in this study were students who tookonline course so the researchers chosequestionnaires for data dissemination, in this studyrespondents gave good feedback so that researcherswere able to process data easily. It could beconcluded that the input and opinions of therespondents showed that the improvement in xyzuniversity online learning needed to be doneespecially in some aspects of the HE aspect.In Indonesia, the application of UXmeasurement by using UEQ had been done for elearning at University of Indonesia which aimed tofind out the UX from the users of learningmanagement systems (SCELE) in the ComputerScience education, it was done by UX measurementthat used multiple methods and the results were1080

Journal of Theoretical and Applied Information Technology28th February 2019. Vol.97. No 4 2005 – ongoing JATIT & LLSISSN: 1992-8645www.jatit.orgused as recommendations for improvement systemfor e-learning management at University ofIndonesia. In this study, the main goal is same,namely to find out the opinions of users in usingxyz university online learning in order to improvethe performance of online learning. The differencefrom this research is the use of the second method,if research at University of Indonesia chose thesecond method in the form of direct interviews withthe users, in this study the research will usequestionnaire as the media, with expectation to beable to give the respondents plenty of time andopportunity to give their opinion so that theobtained-results are maximal. The contribution ofthis research is expected to be a reference formeasuring user experience in the future studies.The scope of user experience research isnot on

user experience questioner (UEQ) as a quantitative study and evaluation heuristics (HE) as a qualitative study. The data retrieving in the measurement of user experience is run by distributing questionnaires to 100 students of the online-learning program in University xyz. Sub