IFN650 Business Process AnalyticsAssignment 2: Marking Criteria
IFN650 Business Process AnalyticsAssignment 2: Marking Criteria
PART A: DISCO
Criteria High Distinction (7) Distinction (6) Credit (5) Pass (4) Fail (3)
Process Discovery
(5 marks) r Comparison of different granularity models captures many significant differences.
r Highly accurate assessment of extent of case variants present in the log.
r Explanation of high number of case variants on representative modelling shows very high level understanding.
r The presentation of this section is done professionally. r Comparison of different granularity models captures some significant differences.
r Accurate assessment of extent of case variants present in the log.
r Explanation of high number of case variants on representative modelling shows high level understanding.
r The presentation of this section requires improvement. r Comparison of different granularity models captures some differences.
r Accurate assessment of extent of case variants present in the log.
r Explanation of high number of case variants on representative modelling shows good understanding.
r The presentation of this section requires much improvement. r Comparison of different granularity models captures minimal differences.
r Generally accurate assessment of extent of case variants present in the log.
r Explanation of high number of case variants on representative modelling shows some understanding. r Comparison of different granularity models does not capture significant differences.
r No explanation of high number of case variants on representative modelling, or explanation shows little understanding.
Mark Indication: 5 Mark Indication: 4 Mark Indication: 3 Mark Indication: 2 Mark Indication: 0-1
Process Comparison
(5 marks) r Thorough and detailed comparison / contrast of case groups in terms of process behaviour and performance.
r Observations point out multiple, significant differences.
r The presentation of this section is done professionally. r Detailed comparison / contrast of case groups in terms of process behaviour and performance.
r Observations point out some, significant differences.
r The presentation of this section requires improvement. r Detailed comparison / contrast of case groups in terms of process behaviour and performance.
r Observations point out some differences (not necessarily the most significant differences).
r The presentation of this section requires much improvement. r Reasonable comparison / contrast of case groups in terms of process behaviour and performance.
r Observations point out some differences (not necessarily the most significant differences). r Case groups are not adequately compared.
r Observations point out limited number of, and/or non-significant differences.
Mark Indication: 5 Mark Indication: 4 Mark Indication: 3 Mark Indication: 2 Mark Indication: 0-1
Filtering
(5 marks) r Correct filter options selected and are properly applied.
r Overview / statistics provided.
r Correct answers to all five questions. r Filter options selected are generally correct and are properly applied.
r Overview / statistics provided.
r Correct answers to nearly all of the five questions. r Filter options selected are generally correct and are properly applied.
r Overview / statistics provided.
r Correct answers to some of the five questions. r Filter options selected are somewhat correct.
r Overview / statistics provided.
r Correct answers to some of the five questions. r Filters not properly applied or incorrect filter options used.
r Overview / statistics not provided.
r Answers to queries not correct or not provided.
Mark Indication: 5 Mark Indication: 4 Mark Indication: 3 Mark Indication: 2 Mark Indication: 0-1
IFN650 Business Process AnalyticsAssignment 2: Marking Criteria
d
PART B: PROM
Criteria High Distinction (7) Distinction (6) Credit (5) Pass (4) Fail (3)
Process Discovery
(10 marks) r Best possible process models discovered using each of the required modelling tools developed with screenshots of each model provided.
r Discussion shows a very high level of understanding of analysis results.
r The presentation of this section is done professionally. r Representative process models discovered using each of the required modelling tools.
r Discussion and insights show a reasonable understanding of analysis results.
r The presentation of this section requires improvement. r Good process models using each of the required modelling tools developed with screenshots of each model provided.
r Limited discussion provided.
r The presentation of this section requires much improvement. r Reasonable process models using each of the required modelling tools developed with screenshots of each model provided. No discussion.
r Poor models, or not all required modelling tools used.
Mark Indication: 10 Mark Indication: 7-8 Mark Indication: 6 Mark Indication: 4-5 Mark Indication: 0-3
Conformance & Performance Analysis
(10 marks) r Conformance & Performance analyses conducted according to requirements.
r Discussion and insights show very high level understanding/interpretation of analysis results for all analyses.
r The presentation of this section is done professionally. r Conformance & Performance analyses conducted according to requirements.
r Discussion and insights show reasonable level understanding/interpretation of analysis results for all analyses.
r The presentation of this section requires improvement. r Conformance & Performance analyses conducted according to requirements.
r Discussion and insights show limited understanding/interpretation of analysis results for most analyses.
r The presentation of this section requires much improvement. r Conformance & Performance analyses cover some of required elements.
r Discussion and insights show low level of understanding/interpretation of analysis results and cover some analyses. r Limited or incorrect discussion of analyses
Mark Indication: 9-10 Mark Indication: 7-8 Mark Indication: 6 Mark Indication: 4-5 Mark Indication: 0-3
GROUP MARKS: /35
PART C: BPI CHALLENGE LOG EXPLORATORY ANALYSIS [INDIVIDUAL]
Criteria High Distinction (7) Distinction (6) Credit (5) Pass (4) Fail (3)
Process Mining
(15 marks) r Evidence of log exploration and meaningful insights were provided. Justification thoroughly supported using analysis results.
r Multiple tools (minimum 3) and multiple algorithms were demonstrated/used.
r Data-informed recommendation is valid, reliable and actionable.
r The presentation of this section is done professionally, and aligned with the rest of the report. r Insights were provided. Justification well supported using analysis results.
r More than one tool was used and the algorithms are correctly applied.
r Data-informed recommendation is valid, but assumptions are not reasonable.
r The presentation of this section requires improvement. r Insights were provided. Justification reasonably supported using analysis results.
r Only one tool was demonstrated/used.
r The provided recommendation is not actionable/not valid.
r The presentation of this section requires much improvement. r Insights were not meaningful.
r Poor justification of insights has been made.
r Unreasonable assumptions are made. r Not attempted or sufficiently addressed.
Mark Indication: 14-15 Mark Indication: 11-13 Mark Indication: 8-10 Mark Indication: 5-7 Mark Indication: 0-4
Learning Outcomes Marks Awarded _____ out of 50 marks
Analyse existing processes by discovering process models automatically and assess the performance using process mining techniques.
Recommend and justify data-driven process optimisation opportunities.
Work efficiently in a group and individually.
IFN650 Business Process Analytics
Assignment 2
Key Information
Type Group Assignment
(3-4 students)
Period 2023/SEM-1
Due Date Wednesday 31 May 11:59
Deliverables Required
Professional Written Report: 50% (incl.15% individual marks)
Background/Overview
In this assignment, you are required to demonstrate your understanding of process mining by using different tools and techniques to analyse execution data. In answering specific questions about the data, you will apply various process mining techniques in order to interpret event log(s) and draw meaningful evidence-based insights from the data.
Part A Part B Part C
Disco 15% ProM20% Exploratory Analysis
(Multiple Process Mining Tools) 15%
Process Discovery 5% Process Discovery 1&2 10% Improvement Recommendations 15%
Process Comparison 5% Filtering 5% Process Conformance & Performance 10% Deliverables
Written Report
You are required to submit a written report that answers the assignment questions together with screenshots and explanations to illustrate the process mining results. Key findings/insights should be discussed in depth.
Your report should be a maximum of 15 pages of content for Parts A and B together with a maximum of 5 pages of content for Part C for each member.
Part A: Disco Analysis (15%)
You are to use an event log BPI_Challege_2017.xes. It is a real-life event log publicly available for process mining analysis. Unless otherwise specified, the complete, unfiltered, original log should be used to answer each of the three questions using the Disco software.
Analyse and interpret the following process models generated by Disco.
Compare and contrast two process models (maps) one generated using the setting: 100% activity and 100% paths and the other generated using the setting 50% activity and 50% paths. Explain why we should not use a model with 0% paths to understand process behaviours.
Investigate the case variants detected from the log. Overall, how many case variants are present in the log? Report on the top five (5) most frequent case variants and their respective frequencies. How much of the log do these five (5) case variants cover? Investigate the variants with low frequencies. How many variants only have less than 10 cases? Explain the implications of a high number of case variants when you try to generate a representative process model.
Compare their process behaviours (i.e., Are the two process models quite similar or very different? Observe the activities/paths/rework loops etc) and performance (i.e., throughput times, # cases, bottlenecks) of two groups of cases. Describe your observations.
Group A: Cases for Business loans (i.e., attribute LoanGoal value Business goal).
Group B: Cases for Home Improvements.
Sequentially apply the following filters to the original log.
1) Only keep cases whose behaviour is shared by at least 50 cases;2) Filter out (discard) all cases that were not started and completed between 1 April 2016 and 31 Dec 2016;
3) Only keep the cases with W_call after offers being the last event.
Show the overview screen with the statistics of the filtered log.
Using the filtered log, answer the questions:
How many cases are there in the log?
What is their mean duration?
How many variants are there in the log?
How significant is the problem of rework for this process?
Are there any bottlenecks detected? If so, which activities/paths are involved?
Part B: ProM (20%)
Use the event log RequestForPayment.xes for Part B. This log is provided as part of BPI Challenge 2020. The data comes from the travel reimbursement process at TU/e. This log contains Requests for Payment (should not be travel related): 6,886 cases, 36,796 events.
Please undertake an exploratory analysis of the log first. You can filter the original log if you wish when answering each of the four questions using the ProM framework (all required plug-ins are available in ProM Lite).
Discover the process models using the Alpha Miner and the Inductive Miner (Petri Net) algorithms (90% paths for inductive miner). Compare the similarities and differences between the two Petri-nets discovered.
Discuss the different visualisations (process tree, BPMN) available from the Inductive Miner algorithm (process tree). Write down a brief process description based on the discovered BPMN model.
Using the Petri nets models discovered by the Inductive Miner algorithm (at 90% paths), replay the log (Replay a Log on Petri Net for Conformance Analysis plug-in). Explain how well the models describe the process behaviour seen in the log, including a discussion of the following trace fitness metrics.
Does the model completely fit the log?
If not, how many cases fit the models and how many do not?
Where are the problems for the non-fitting process cases?
Analyse the process using the Inductive visual Miner and identify potential bottlenecks and deviations. Note: Inductive visual Miner is also made available as part of QuickVisualiser: http://leemans.ch/leemansCH/quickvisualiser/Part C: BPI Challenge 2011 Log Exploratory Analysis (15%) [Individual Task]
Use Hospital_log.xes for Part C. This is a real-life log, taken from a Dutch Academic Hospital. This log contains 46560 events in 824 cases. Apart from some anonymisation, the log contains all data as it came from the Hospital's systems. Each case is a patient of a Gynaecology department. The log contains information about when certain activities took place, which group performed the activity and so on. Many attributes have been recorded that are relevant to the process. Some attributes are repeated more than once for a patient, indicating that this patient went through different (maybe overlapping) phases, where a phase consists of the combination of Diagnosis & Treatment.
van Dongen, Boudewijn (2011): Real-life event logs - Hospital log. Version 1. 4TU.ResearchData. dataset. https://doi.org/10.4121/uuid:d9769f3d-0ab0-4fb8-803b-0d1120ffcf54[Individual Task] Imagine that you are asked to present one data-informed process improvement recommendation to stakeholders. Your task is to extensively analyse this process (guided by the stakeholder questions) and provide one improvement recommendation based on these analysis insights. Please note that this is quite a complex process. You can filter the log as you see fit. You must explain your assumptions, and filtering rules and justify your recommendation using the process mining results (including screenshots).
You are encouraged to investigate multiple tools (e.g., Disco, Celonis, ProM Lite, Quick Visualiser, Apromore, etc) to derive your insights.
Group Work
The assessment tasks are to be completed in a group of three or four students. Each student must individually register in a group on Canvas. All group members are expected to contribute equally to the assignment deliverables (Parts A and B) and conduct Part C individually.
Submission
Your report must be submitted online, using the link on Canvas.
Your team should submit one assignment, uploaded by any one member of your team.
Format of Submission
You must submit a single .pdf file (named IFN650_Assigment-2_Group-XX.pdf), containing part A, B and the individual parts C of each member. The formatting of the whole report must be consistent.
Late Assessment Policy
Assignments submitted without an approved extension will not be marked and will receive a grade of 1 or 0%. You can apply for an automatically approved 48-hour extension, or if special circumstances prevent you from meeting the assignment due date, you can apply for a longer extension. If you don't have an approved extension you should submit the work you have done by the due date and it will be marked against the assessment criteria.
