Step 6 : Gauging the impact of the test
Time : 
ongoing for the whole project.
To be mobilised : Project point of contact, procurement manager for stock monitoring, people at reception and on room service for monitoring customer feedback, departmental heads for monitoring collection of various data.
Gauging the impact of the project means analysing the effectiveness of the test, and being able to actually draw benefits from its endeavours and successes, and to know precisely about its methods of improvement. We suggest a classic and robust 3-stage method: i/specify your indicators against your objectives, ii/collect the data required and iii/interpret the results. You can easily specify your indicators at the same time as your objectives (see 1.1 Clarify objectives) to save time!
6.1 Specify key indicators
Refer to your initial objectives again and turn them into big questions, such as:
- Did the pilot project enable a reduction in Single-Use Plastic (SUP) use without leading to transfer of environmental impacts?
- What are the costs and benefits of the process? At what point in time are the alternatives amortised?
- Did customers and staff stick to the process and accept the changes?
These important questions will enable key indicators to be defined (Key Performance Indicators, KPIs), and make it possible to choose which data to collect depending on this, and to be able to set up the data collection process.
Caution! You need to choose a year and a reference period: all these questions pertain to a starting point. You will see that this will become very clear in the
next section!
THE CASE OF THE INTERCONTINENTAL MARSEILLE : The pilot’s KPIs
Concerning environmental matters:
- average mass of SUPs avoided over test period
- percentage reduction in consumption of SUPs over test period
- average environmental impact
Concerning economics:
- spending on test implementation
- savings achieved for test period
- return on investment
Concerning social acceptability:
- perception by staff
- perception by customers
6.2 Identify and collect the data required
You have defined your KPIs, so you now need to analyse them to identify i/how you will calculate them, ii/data that you require, and iii/the data collection method.
Calculation methods are specific to the chosen indicators (see details of the case of the InterContinental Marseille – Hotel Dieu in standard model #3). However, there is a common point for all indicators: the need to specify a reference period (situation before rollout of alternatives) and a test period (during rollout of alternatives). Data gathered for the test period will be compared with those from the reference period to see how the situation has developed.
A POINT TO LOOK OUT FOR
While altogether laudable to want to carry out this process as precisely as possible, we have decided on a “coarse mesh” analysis taking into account the constraints of time and budget. This continues to be robust from a global methodological viewpoint. We know and recognise the approximations we have put into effect and the hypotheses we have put forward. You are free to add finer points, completeness and specific cases!
Choose the reference period and test period:
- What are the dates of your test period? Though your alternatives can be set up gradually, we recommend simplifying and conducting your analyses over the same period for everything.
- What is your reference period? The simplest thing is to take the similar period for the previous year if the situation has not turned it into an atypical year.
In terms of data for collection, although this is dependent on your KPIs, here is a list of the must-haves:
- the mass of plastic of SUPs and alternatives for calculating the mass of SUPs avoided;
- Visitor numbers to the hotel for the test period and reference period: this will enable you to calculate the consumption rate of an SUP or of an alternative, the feedback rate of customers etc. ;
- the quantity of SUPs and alternatives ordered for the test period and the reference period: this will be useful for calculating the mass of plastic avoided, the consumption rate etc. ;
- the unit cost of SUPs and alternatives: this will be vital for estimating savings or overcosts caused by the changes;
- customer and staff feedback: it is vital to collect their experiences and to go beyond what the project team perceives.
Some data may be collected before, during and after the test phase. For example, the mass of the SUPs can be weighed or known (via technical sheets) from step 2. In all cases, we recommend listing them and centralising them in a standard model #3 (see table), to facilitate collection and analysis.
Allow time to check, KPI by KPI, the calculation method and the data required. By doing so, you will be able to picture the way of collecting data: does my purchase and stock management software enable me to have this information? Will my complaints system suffice for knowing about any malfunctions? Do I have access to webometrics data for finding out about the number of clicks on my website?
OUR TIPS
Always come away with thoughts about the question and about the initial objective: it is so easy to become lost in data and indicators! Methodological rigour and precise information are valuable, but stay on course for the KPIs; this alone makes for a great deal of work!
THE CASE OF THE INTERCONTINENTAL MARSEILLE : The environmental KPIs, from the key question to the calculation methods and necessary data
THE CASE OF THE INTERCONTINENTAL MARSEILLE : The monitoring-evaluation of social acceptability
To know whether the changes were accepted socially, we have looked at two targets: the staff and customers of the hotel.
Monitor and assess the acceptability of changes by staff:
Start by involving your colleagues as early as possible and base things on their practices and needs as proposed above (see 5.1 (inform and train staff). Then, to formally collect their feedback at the end of the test, you can use the following indicators, developed by InOff Plastic and used at the InterContinental Marseille – Hotel Dieu:
- Overall perception of alternatives
- Degree of change engendered by establishing the alternative (department heads)
- Degree of change engendered by day-to-day management of alternatives (staff)
- Degree of practicality/difficulty engendered by the alternatives
- Time elapsed in establishing alternatives, time taken to set up alternatives vs. time taken for staff to set up alternatives before setup was complete
They were assessed using a questionnaire with a grading scale (eg. 0 = no change, 10 = major changes) and quality-based feedback collected at the monthly team meetings.
Monitor and assess the acceptability of changes to customers:
Gauging customer satisfaction was done by researchers from the IMSIC using existing tools and others developed for this purpose:
- questionnaires: the hotel’s general satisfaction questionnaire, and a specific questionnaire accessible by the QR from the communication signage
- from a webometric analysis: opinions left on booking websites and from the number of customers using a QR code from signage created for the project
Conducting interviews would have been ideal, but this system was too cumbersome and time-consuming to set up.
6.3 Interpreting results
So now is the time, armed with your experience and your KPIs with regard to environmental, economic and social aspects, that you’ll be able to build an opinion with regard to the effectiveness of the test. Analysis of results enables a conclusion to be drawn with regard to alternatives to retain conclusively, to roll out more widely in different areas of the hotel, to adapt (from the point of view of protocols, communication messages) or to redesign more drastically due to the environmental impact.
In the case of the InterContinental – Marseille Hotel Dieu, it made it possible to get rid of concerns about certain re-employable alternatives that initially appeared difficult to set up for reasons of hygiene (eg. using and washing sugar bowls), acceptance by customers (eg. cancelling room amenities and making them on request) or time spent by staff (eg. re-filling water bottles). Other results, meanwhile, went against hopes (eg. sweets made available loosely to replace separate bags).
CLOSEUP ON : Interpreting the lack of feedback from customers
Of the 15,000 customers received during the test period, only 307 completed the satisfaction questionnaire and only 4 of these gave a negative opinion (all with regard to a lack of bathroom kit). Information and communication science researchers concluded from this that the micro changes brought about by the reduction in SUPs in the hotel went unnoticed among customers.
Éric Boutin and Daphné Duvernay (IMISIC)It’s not a bad thing that the hotel succeeded in changing behaviour with regard to plastic use in a way that was “invisible” for its customers. We know about peoples’ reticence towards change which is readily expressed. Having succeeded in changing things without leaving oneself open to criticism is therefore in itself a positive aspect.
OUR TIPS
View your operation as a process of ongoing improvement and as a repeated process! You already know what your quick wins are, as well as your disappointments and your levers for improvement, and you should be commended for that! Regularly assessing your KPIs will enable you to adjust your output decisions for SUPs. And generally speaking there are two pathways for taking things further: i/returning to the diagnosis stage and dealing with SUPs that you deemed non-priority and ii/improving management of your waste in different spaces (separation,
collection, recycling).
