CMAF FFT: Common Forecasting Methods and The Mindset Behind Forecasting

right hello everyone it is Friday it is 2m in the UK which means that we start our Friday forecasting talks my name is Ian and I'll give you a brief introduction of

uh the center that organizes the talks and here is a slide with all the center members current Center members that we have at the moment so one of

those members will be presenting today that's Stefan colassa but you see that we have uh well 12 12 um

academics that have a variety of interests and based on our interests we provide a variety of services we spoke short courses consultancy summer projects we help in developing software

to companies and we have expertise in variety of areas including marketing analytics supply chain forecasting uh Inventory management forecasting and planning and so

on um before we move to the further discussion of how you can get in touch with us and so on I would like to say a couple of words about the survey that is

being organized by our PhD student Carlos Rodriguez under supervision of Sven cron and analina so Carlos is trying to understand what is the state of

forecasting in demand planning and he needs your help if you're uh involved in demand planning in one way or another so we would really appreciate if you could uh answer

several questions in the survey it shouldn't take more than 10 minutes of your time so the survey shouldn't take more than 10 minutes of your time please do have a look at it and there is a

LinkedIn link so if you can share it with your friends colleagues who might find it relevant it would be great because what we are trying to do we are

trying to understand what people use in practice how they use those things and so on specifically in forecasting okay and a couple of words

about the center you can get in touch with us keep in touch with us using lots of different ways we are on Twitter LC sof we are on LinkedIn uh you can send

us an email directly we have an amazing new website uh developed by uh our great intern and under my supervision we have

YouTube channel where we publish these videos plus we are now releasing educational videos and stay tuned at some point hopefully in a month

or so we will have a new video delivered by the very same Stefan colassa who will be talking today and finally we have a landing page of Friday forcasting talks where we

publish all the relevant information to these events if you feel a bit lazy following these links typing them and so on you can scan

this QR code and it will lead you directly to the page with all the options uh related to simf well that's actually it from me so

let's move to our presentation and our speakers today we have Stefan kalasa bahman rest time in tabar and ano Simpson and they will be talking about common forecasting methods and the

mindset behind forecasting all right so well hi everybody uh really delighted to be here my name is bahman and I'm a reader at

card business school cardi University in the UK I'm I'll pass it to to Stefan H sorry uh thank you ban thank you everybody for being here I'm Stefan

colassa I'm a data science expert at ACP in Switzerland um Eno over to you uh good morning everyone great pleasure to be here thank you for joining my name is eno zson and I'm a

faculty member at the University of Wisconsin Madison um Anna would you like to provide like a background to the book before we go through the presentation

maybe yeah sure um so the presentation today is based on on our book which has just been released in in print and it's available for free

online um you can see the the link over there and that um really this goes back almost 10 years ago um when you know I I

kind of observed that there were plenty of forecasting books out there and uh very good technical books but um there's a large audience that uh was

particularly relevant for Executive Education um of people who who are not Technical and who don't necessarily have a technical background but who have to

deal with forecasts and um so I felt we needed a book which is more written for that audience so I approached Stefan uh probably somewhere around

2014 uh and asked him whether he he wanted to write a book with me and and that became book called demand forecasting for managers which has been out for a while um really sort of

targeted at at a managerial audience and um roughly in 2020 bahman approached us and and said well I I really like your book which you know it's always good to

hear as an author and um and basically said we need to do two things we need to update the book because a lot of things have happened in the field of forcasting

uh since we wrote the the first version of the book um you know machine learning has become a lot more useful and uh we um really have to add a couple of topics

and uh second we have to make it available for free and so um as authors you know you you don't really earn a whole lot of money from your book anyway so you're like okay

great let's make it available for free so you can share it you can use it in your um faculty can more easily use it in classes uh Executives can share it

with with their teams right so um that led to this version of the book and and really the key idea of the book is to again make forecasting accessible for an

executive audience right there's a lot of people out there who don't have Technical Training but have to make decisions based on forecasts and there's so many things that can go wrong um with

that so we wrote the book with that audience in mind bahman next slide wonderful uh thank you Eno uh So based

on the the book actually we have um this agenda for today uh we start by talking about why we do forecasting in the first place and why it is important for

planning and decision making processes and then we uh go through different steps in the forecasting workflow to produce the forecast to inform the

decisions uh we present a simple example of how forecast are our produce and communicated as well uh and then we talk about an important aspect of the process

which is about the data and why we need to know data and to do what with actually that um and following that we look at the some aspect of data quality

that are important for forecasting and then we discuss about um a range of different forecasting methods we don't go into detail of the method we just show you what are the different range

from simple to more uh complex and then also discuss our recommendation and how to build forecasting models uh following that we look at the forecast quality

evaluation and also discuss what uh attributes you need to have uh for uh like an ideal forecaster in your team um so in addition to like the technical

part of the process we also discussed the organizational part of it and then finally we uh cover some things in terms of when for forecasting May

Fail we start the book with um the presentation today here with with sort of just a a bit of a sense for why forecasting is important and and maybe

for this audience this is this is sort of Fairly obvious right but I I've run into this quite a bit in my career doing Executive Education Consulting where where people will say why do we even

bother with forecast in right forecasts are always wrong um can't we just switch to a business model where we don't need

forecasting and and that sounds very tempting um but it is I think utterly impossible right uh someone in your

supply chain will have to forecast sure you can you know move to sort of uh quick response and and faster uh uh uh lead times with your suppliers but that

usually means that that puts a heavier burden on your suppliers to do the forecasting um you can move to a pull system but you know that usually means you have sort of an implicit forecast of

things becoming relatively stable so the there is a I think a a need in every organization to do forecasting and that's important to emphasize there's

really no way around it you might as well uh explicitly tackle the pro uh the problem right so um somebody in the

supply chain has to forecast and um you know forecasts are in incredibly important um for any decision making any planning and but forecasting doesn't

have to be complex right I think we have seen an explosion and in methods in forecasting um you know particular in the past five years I think the the the complexity of these methods has

increased significantly um but very simple methods actually work quite well I think I think there's a lot of lwh hanging fruit where um you know if you don't want to tackle

complex uh forecasting methods you can get away with relatively simple methods and they tend to already perform um quite well and on the plus side you know

if you if you do forecasting well right there's tremendous benefits to it um the better you forecast in some sense the less firefighting there is in your

organization and your supply chains um you know you you of course get immediate benefits such as higher service levels lower Safety stock requirements overall

less friction in operations but I think um maybe the two most important things if you do forecasting and sort of the related sales and operations planning

right um it's ultimately the key aspects over here are organizational aspects it's difficult to do but once you do it right uh it's hard to copy I think it's

it's quite a competitive advantage and um you know overall it allows your organization to really focus on the

things that drive value right bad forecasting leads to a lot of frictions that remove your attention from what really matters to just kind of

firefighting and and things like that so forecasting I think is incredibly necessary um for creating any kind of

predictable and um stable oper ation next slide uh all right so generating the forecast to inform the decision that

discussed in the previous slide uh is a is typically uh iterative process that requires um several steps that are highlighted in this flowchart so this U

forecasting workflow always starts with identifying a decision that requires a forecast this uh this seems obvious but sometimes s actually we forget about

this and uh we are too enthusiastic and we rush into the model Building without understanding really why we do the for why produce the forecast in the first

place um uh and then uh we generate the forecast and we don't know how to use it later so I think it is really important to spend some good time in this first

step to identify decisions um typically we have one or many probably reasons or decision that require the focus so uh we need to identify them first and then

following that um basically what we identify in step one uh help us to determine the forecasting requirements which generally includes what is the the

thing that you want to forecast for how long in advance you want to forecast it what is the granularity that you need in terms of temporal granularity but also cross-sectional

granularity uh what is the frequency of producing the forecast how often uh you need to produce the forecast test and so on uh so in following that in step step

three we uh talk about Gathering data information and here uh we typically talk about four

uh type of data first of all we have um past historical data for the variable that you want to forecast and then we talk about also the past and future

value of deterministic predictors so these are predict their value are known in advance like promotion or um public holidays and the third type of data is

what we called um stochastic predictors so these again we need the past and future values of them depending on again how we use them but these are predictors

their future value are not known and we may need to estimate them and finally we also uh need the collective expertise or Judgment of uh key people in the

organization as well so once we have the data that we need um and typically in any organization the row data is never

in a format that it is ready for um forecasting or anal data analysis in general so we need first to put it in a format that is uh required for the

analysis but also we need to check the different aspect of the data quality here as well including like missing values duplications the accuracy of the

values and and so on uh and following that we provide different sort of uh plots or data visualization but this step four

and five they're sometimes happening actually simultaneously because we can also use data visualization to check for the quality of data sometimes but in

step five you are looking for sort of systematic patterns in our data that could help us to um build a model but also uh this is where

we need uh maybe domain knowledge to um understand basically the data and in Step six we have to uh choose from like

a a bunch of different fasting methods so which method actually we have to use um and again this depends on many factors um we could also choose a method

from very simple one to go to a more uh complicated one um and then we'll discuss it actually a bit later in detail um and we following that we have to train the model on the data and then

produce the forecast which could be in the form of Point forast prediction intervals distribution or the sort of quants if you want to extract from the distribution and then we have to

evaluate the quality of the forecast and here the quality may have different dimension as well including of course forecast accuracy business impact but

also other aspect of quality like uh computational time or um interpretability for instance so this

sort of steps are are you may go through them multiple times and uh following that when we are happy with this we can now communicate the forecast to the

audience and here we may ask ourselves two questions so what we have to communicate but also in what uh format we have to communicate it and typically

in the workflow uh in the last step or before maybe communicating with different audience or simultaneously we have to also um adjust the forecast

sometimes uh this may happen when we have some new information that we didn't have it before producing the forecast just they came out or sometimes you may think that some information are actually

missing so the model didn't capture that information but again the question is if we now know them we could include them in the model so why not in instead of

um adjusting the forecast why don't we just include them in the model building um so these are basically all the steps we need to go through when it comes to

producing the forecast uh and the repetitive step that we're talking about we we're a big believer in simple examples and again that that's partially

because of the audience we're addressing in the book and um uh this this is sort of a an example where um we introduced forecasting methods in the book and and

really sort of we want to make a couple of of important points over here so for example the very first one is um

that we um that that point forecasts by themselves are fairly useless um it's still very common in organizations to Just Produce Point forecasts right but

if you do that um you have have sort of two problems with it the first problem is um you create the illusion of certainty where there really is no

certainty right a forecast is not really uh a description of exactly what's happens a forecast is an estimation of a probability distribution ultimately and

um and that has to be communicated right if you don't communicate it um Everybody in the room will will think you know will will make their own judgments about

how certain forecast actually is right and um or some might even think that the forecast is is quite certain where in fact there's a lot of uncertainty around it right so um you need to think about a

forecast really as an estimate of a probability distribution so if you consider the example here on the right hand side it's a um simple time series right if you look at the sort of um the

graph on the upper right hand side um this is a um you know a a simple random walk and um you know the the first thing that you do if you forecast is if you

get a you know if you look at the middle graph right you get the sort of the center line over there that would be a forecast for the future and um this is another key learning I think

that we we want to illustrate is a lot of people will think of that that line of forecast right if you have a random walk underlying the data if you have no Trend or

seasonality um your forecast your set of forecasts for uh the next periods will essentially be constant and this is incredibly counterintuitive for people

right people expect forecasts to vary but forecasts are ultimately noise reduction tools right they should the the series of forecast should look a lot less variable than the series of data

that preceded it um so again this is sort of I think a key learning from any Executives is sometimes a series of forecasts is is just a flat Series right and you can see that in the in the

middle graph right the the point forecast that's sort of the dotted line in the center that's just um kind of the uh the sequence of forecasts going forward and then around that of course

as I said we need to to produce um measures of uncertainty we have prediction intervals and you can see they start narrow and they widen as you predict further into the future um and

again that's a an important aspect about forecasting too right the more you predict into the future the higher is the likelihood that the underlying uh aspects of your of your

data generation kind of change right in this case that the the level of the time series of the random walk shifts around and that increases um the prediction

interval and on the the lower part of the graph we added a predictive density right which is um in some sense the most complex way of representing uncertainty

um but that gives you kind of um you know very clear um graph of the underlying distribution right so with prediction intervals you ultimately um

you kind of have a usually something like a 90th on uh 90% of the distribution that you cover right so you you end up showing best case and worst

case scenarios that aren't really best case and worst case scenarios right the the density can extend that so if you want to communicate that even further you can use a predictive density um but

again at the very least um communicate prediction intervals um make sure that that Executives understand um that there is uncertainty around forecast and make

sure that they understand that point forecasts can sometimes just be a straight line if you don't have any indication that there is Trend seasonality or any other causal drivers

then that's exactly what a sequence of forecasts of Point forecasts will be like bahman great so possibly one of the most important

steps when it comes to building a forecasting models that really work in Works in practice is to understand uh your data and of course here in

forecasting we're talking about understanding the time series so uh here for instance we can use time series uh Graphics like a simple time plot to

start with to plot the data and the idea is of course to understand what are the sort of systematic patterns in the data including Trend and seasonality possibly multiple

seasonality if there are some um unusual u values like very high or very low we need to understand those sort of values so I would say in in Real World

um it is very rare actually to have a Time series that that is not affected by different events and this is where you know having the domain knowledge is fundamental it helps to understand the

the different sort of uh events that affect the the time series observations and uh also by by having those information we can go and collect maybe

relevant data that could be helpful in building the model it is also important to look at uh data from different perspective so or or using basically

different sort of data visualization as well to uh to understand the data this is an example of ambulance data for instance so in the middle we have a a

Time plot uh of hourly data so of course we can see some sort of systematic patterns but also some uh observation that are very low and very high possibly

related to um public holidays uh but if we look at this data from another uh aspect which is um Illustrated in this plot in the bottom where we have the

hour of the day uh in x-axis and then different lines Cor responding to different day of week we can see actually something really interesting happens that we couldn't see in the time plot and this is where for instance we

see the ambulance demand between midnight and um early morning for Saturday and Sunday is quite different from the rest of the week and if you look at the other side of this it means

from like uh 8:00 p.m. towards midnight

we see that the demand for Saturday and Friday is uh again different uh for that um you know of the day so there is an interaction between the different day of

week and hour of the day and also we can see a sort of interesting pattern happens in the daytime so this highlights how important it is to uh to

understand uh the data and also the systematic pattern that may exist that help us to build a model and of course if we are dealing with a problem where there is no systematic pattern or at

least less systematic pattern the time series is dominated by random or noise then um it might be really hard to produce a forecast that would be helpful

uh to inform decision making all right so now we've seen everything that's important about data and the key issue here is that data is often of doubtful and dubious quality

and so honestly data quality is hugely important in forecasting and that is sometimes not uh appreciated quite as much as it might be and by data we mean

both the focal time series you want to forecast if there is a focal if there is a Time series if we want to forecast a cult start and new product and we don't have a Time series but then we have

predecessor products or similar products so their quality is important and on the other hand it's also any causal drivers so if we have a promotion or something

already mentioned that one uh then it's important to know when past promotions happened what kind of promotions they were what their features were and when they are going to to recur in the future because if we just know there were

promotions at some point in the past but we don't know anything else then uh we are we have a problem essentially and so honestly in our experience uh it's often

the case that better data and better understood data it's often more important than fancy modeling and uh that's possibly not uh the impression

you'd get from academic research and forecasting where people will take a data set that they got somewhere and they'll just assume the dat as given and their understanding of the data as given

and then they'll start doing the fancy modeling now very often you just see that if you are able to get a better grip on the data itself then you can

improve on many of the more complex methods okay so how do we understand our data better well that's always a question of having the denain knowledge so understanding where the data comes

from uh what we just saw was was ambulance data so it's it's important to understand what happens at in the UK on Friday evening on Saturday evening that might cause uh more calls for an

ambulance so there's domain knowledge for you and that also tells you what to look for what's going to be important what kind of data is probably what kind of causal drivers are probably driving our times here so that's what you need

to look for that's how you understand what may be a problem problem in your data and that will help you understand uh where and when to look for problems

and how to solve those problems if if you have problems and uh how do you deal with those problems um you can always try imputing missing data I personally

am not big fan of imputing data data imputation because it kind of simulates knowledge and understanding and certainty that we actually don't have so if there is um anything that's

problematic then you can try to correct it or again impute it or you might also be it might be possible to just use a different predictor and say we're not sure about this data point so mark it in

a causal way and let the model um figure out how to deal with that can you skip to the next slide please thank you all right so now we've good data or we all

hope that we have good data now we are ready to start building method models and figuring out methods using forecasting H here's just a very simple

plot that kind of conceptually gives you the the trade off uh between ban could you please be so kind to turn off your your mic thank you so much lovely thank

you uh it gives you a bit of a trade-off between model complexity and interpretability complexity does not necessarily mean that more complex methods are more accurate it just seems

that they're more complex and we start looking at the top left at things like the seasonal mean so just take the historical times here take the overall historical mean and just forecast that

out as a flat line or a naive forecast take the last uh data point and forecast that out a season and naive take the last data point from the same season and

forecast that out so if you want to forecast the next January then just take the last January observation of forecast thata or take the average of the all the historical January observations and use

that as a forecast for next January it's extremely simple and often works surprisingly well and it's very hard to beat by more complex methods especially if your input data is

iy well beyond and that should always be used no no no don't don't don't don't go ahead thank you sorry okay so one once we have that we can start looking at

more complex methods uh like exponential smoothing ARA cross method for uh for for intermittent demands those are also workhorses for forecasters and everybody knows them and that's the first thing

you learn about in forecasting and they're they're available in so many different implementations and for any software that you might care to use those Sil can't model caus effect

effects or predictors so for those we need things like multiple linear regression which is still simple and many people have come across a regression at least in their college

days and beyond that you go into the more modern more complex methods like tree based methods random Forest boosting and then neural networks and deep learning and there's so many

different architectures and methods for all these things especially for deep learning you can combine them and use boosting with deep learning and you can

combine things using meta Learners and you can do lots of things they're less interpretable they usually take a much longer time to fit uh they're more

expensive in terms of you may need to really hire cloud services to actually run to fit those models you might may need to hire expensive data scientists

and perhaps some of those expensive data scientists are in this call here and uh you have to pay for those people to fit those more complex methods whereas it's simple historical mean you can do that

in Excel or SQL or what have you and that's actually pretty simple and pretty cheap and of course on top of all this we always have human judgment humans

will always adjust the forecasts if they get the chance to do so sometimes it makes sense sometimes it makes less sense ban can you click please thank you

so much so how do we deal with all these different models what's what's a good workflow for actually building forecasting model and here's something you would propose we always recommend starting with the

simple methods like as we said historical mean historical quantal last observations that's a na forecast they're very simple to build they're very fast and they should always serve as a benchmark if you can't improve on

the simple mean then building something more complex is just a waste of everybody's time and processing Cycles both human and uh computer processing Cycles once you've built that logical

next step is exponential smoothing or Rema they're implemented they're established tools for those like forecast fa or a smooth package in in our the last one built by by our very

own Eva andonov uh it's I often see questions on Cross validated which is like a QA site for statistics on how to build a model

based on autocorrelation or partial Auto coration plots and honestly I'm mystified by why people would do that because there is so many established

methods that will do that very well for you and unless you're really an expert in these methodologies you should really not try to do that the the established

methods will outperform you and they will be faster so just don't do it by yourself and especially don't follow random internet advice because there's so much random stuff on the internet

where people just build some kind of post somewhere where they start T leaves reading in Auto coration partial Auto coration functions and they give some

kind of advice and you're more confused afterwards than than before so best to really just use an automatic method unless you know what you're doing uh then beyond that if you have

predictors and then you should really look at causal methods like regression which is very simple it's linear so it's kind of constrainted it's not very flexible but that may just be uh what

you need because you should start with the simple method first and make that run and once you're done with that look for the more complex ones always be guided by domain knowledge and

start with the more important drivers first and also in terms of data quality and forecastability so if you have a hugely important driver but you can't forecast that if you know your sales are

driven by the weather but you need a forecast for three months ahead and you don't know the weather three months ahead all you know is the climatology or the seasons that you have then you probably can't really use the weather in

forecasting very well you may need it in cleansing your past data or in fitting your P data but in forecasting it's probably not going to be very useful so use something else well always need to

come to balance the effort in data collection against the Improvement in accuracy if you spend a lot of time collecting data cleansing data understanding data and just get a tiny Improvement in accuracy question really

is is that worth your while and always beware of overfitting It's Always tempting to start fitting more and more complex methods and putting in more and more data and seeing nicer and nicer

insample fits but the sample fits at some point will not get better because you're starting to overfit so be careful about that um beyond that um we can you can

always look at combinations combinations is always a very good tool so that's just an empirical fact that combinations often perform the various constituents

methods or trying to select the single best method so picking the best method uh for any time series that is a commonly use strategy it's often better

to just not do that just use everything combine everything weedly or unweighted uh interestingly enough it's often the case that an unweighted

combination is very hard to beat by finding an optimal combination of setting optimal weights and that's been referred to by the forecast combination puzzle so here's our recommendation

start with a very simple method go on to time serious method look at causal methods and always consider combinations and always keep keep it simple because you can spend a lot of time running down

rabbit holes and uh and doing lots of important stuff and interesting stuff without necessarily getting better forecast out of that B can you click please thank you so much all right so

here's one thing that's actually very dear to my heart that's forecast quality evaluation so now we have our forecast and how do we find out whether that's any good and the problem here is that there's so many things to do here

because there is an entire zoo of forecast accuracy measures depending on what you're forecasting for Point forecast you can have a squared loss you can have an absolute error can have a

percentage ER if you have an interval forecast a quantal forecast so you want to forecast like a value so that yeah you're reasonably sure that uh no more

than 90% of your observations are below that value why would you want to do that well because that maybe your order up to level if you order up to this level then you'll cover 90% of demand so that's an

interval forecast you need to evaluate that there's pinball losses can't evaluate a quanta forecast using a squared loss but a pinball loss that's built for that if you have an inter

interval forecast then you can have an interval score if you have an inter predictive density then you can look to proper scoring rules and these are actually in descending order of

interpretability because at some point in time proper scoring rules really I don't think anybody understands those I certainly don't but they're there to help you evaluate d forecasts the

problem is especially Point forecasts can be actively misleading so if you the mean absolute percentage errors for instance is undefined if you have an a zero in your actual time series because

you'd then be dividing by zero uh your maths are not going to like that and if you have an intermittent or low volume count data time series and the mean

absolute error can be very misleading it might lead you or incentivize you to give an an output that is not the expectation that is highly biased so if you have something that has lots of

zeros it might be best best in scare quotes if you have the absolute error the absolute error could be minimized by a flat zero forecast that's rather

unintuitive and uh that's why I'm always it's always um important to really think about your your time here and what you're trying to do it's always better to figure out what do I want to get out

of my time Ser to want an expectation forecast do I want a Quant for forecast then tailor the forecast accuracy measure to what you want out of a forecast and then work and very often we

see it the other way around people take a Time serious they just take any old forecast accuracy measure because it looks nice like the mean absolute percentage error and then they forecast

out and and then they're surprised uh that an an a biased forecast might perform better than an unbiased one question we of often get and often see

out there is uh using external bench marks so here there's people out there that will sell you um external benchmarks industry standards uh in your

industry the standard error is 30% so if you only achieve 40% you need to hire expensive Consultants to get a better forecast um honestly those benchmarks

those accuracies those industry standards are really meaningless because it hugely depends on what AG granularity you for you forecast on if you forecast on P Q level that's that's going to be

harder than if you forecast on an category or product group level if you forecast on a weekly level it's going to be harder than if you forecast on a monthly level and most of these indust

industry standards don't even tell you which granularity they're thinking of so it's really hard and in addition it's very much depends on what kind of of uh

product you're selling if you sell a staple a very simple stable product if you're selling toilet paper toilet paper demand is very very stable there's no season out there's essentially nothing

happening in there unless you have a pandemic so your forecast is going to be extremely simple if you're selling something else like detergent where you have lots of promotional activities uh

that's going to be much harder to forecast so it really depends on what you're forecasting so you can't just say industry average for for instance drugstore products or household products you could call that houseful products

it's not going to tell you anything so better to Benchmark your processes and make sure that your forecasting process is as good as it can be and then good good accuracy is going to fall out at

the end of it and if you have a wor forecast and you're competitor down the road then that might just be a case of you may not have the best forecast

process or you may just have time series of products that are harder to forecast and it doesn't make a lot of sense to just say well they've got better forecast so we have to do something we

have to fire our people hire more intelligent people hire more credential people to get a better forecast they may not be able to improve on your

particular time series to work on and finally even if you can improve your forecast accuracy that doesn't necessarily mean that you get better business outcomes because forecast

accuracy is nice to have but you can't buy anything out of a lower map or a lower mean squared a can buy something if you have a better inventory position if you have a higher service level if

you have lower inventories that's something that can earn or save you money better a forecast accuracy will not earn you any money and the link between accuracy and business outcome

that's not quite straightforward because a business outcome not only depends on the accuracy of the forecast for the forecast is hugely important but there are so many other things that also tie in here and it's all the supply chain

parameters from logistical units to uh replenishment Cycles to the quality of your suppliers and so on so forth and all these uh get in and all those are

are hugely important too and they may swamp the accuracy bahman one more click please thank you so much there's also a

human aspect so we've been talking a lot about models and methods and Mapes and numbers and theory and abstractness but it's in the end it's all humans that do

the forecasting we've started at the very beginning talking about judgmental adjustments and judgmental impacts on forecasting and we have an enter chapter in the book on uh how to deal with the

forecaster with a human forecaster in the loop so what's the perfect forecaster what should you be looking for when you want to hire a forecasting team and there's tons of vend diagrams

of data scientists floating around out there and our personal one uh really consists of four dimensions so there is um programming of course every data scientist needs to understand

programming whether it's python SQL whatever whatever you want to use you also need to understand statistics you need to understand Randomness variation you need to understand linear

regression you need to understand lots of things there H then the third one is always business or domain knowledge so understanding the domain knowledge if you don't understand the domain then you

will have a hard time figuring out which is the time which are the data points the data that you need which are the predictors that are important and the fourth dimension which is quite as important as the other three is

communication because you need communication at the very beginning of that float chart that we talked about in talking to business people what kind of forecasts you need what is the decision

you want to you want to uh to to use your forecast for uh what kind of data do you have what's the Sur what's the environment what are we talking about that's where you need communication and

you also need communication at the very end the forecaster just built their wonderful lovely forecast and now he has to explain that to the people that will consume it that will actually decide

based on the forecast and the forecaster needs to build trust in the forecast because if the business people do not trust the forecast they'll disregard it and they'll just use their own forecasts

or their own gut feeling or anything and then typically that's not a good outcome for anybody involved so Communication in forecasting is hugely important and

somebody who is a whiz at statistics and programming and understands the domain knowledge but can't explain their forecasts to somebody else they're not going going to be very

useful all right thank you very much I think next is on to somebody else that's en thank you back to me again I always have to follow up with your

four-dimensional V diagram Stefan and I cannot so thank you so much and then as I tell everybody for the next version of the book we'll make it

five-dimensional um all right so I think sort of um one thing which is very dear to my heart is to to always um think of

forecasting not just as a technical and statistical process but ultimately as a as a human process right uh forcasting is embedded into the

organization it is um subject to individual bi biases and it's it's sub subject to the sociological biases that are are inherent in organizations right

so um there's there's plenty of of um companies where um the forecast is

ultimately uh a judgmental one right um most firms like a a statistical forecast is is a baseline uh but it is then

adjusted by decision makers and um it's amazing how prevalent that still is and sometimes for good reason sometimes for

not good so good reason um you know um Paul Goodwin and and Robert files and um Sher debates just have you know put out a working paper where they integrated

sort of what do we know about um judgmental adjustments and the answer is basically they improve accuracy in only 52% of the time so that's a that's a

pretty low number and um and really the reason here are why that's not more effective right is is sort of twofold one is um individual biases and the

other one is sort of organizational biases right so individual biases are are really um um you know what we know about human judgment and decision- making right so there are things like

anchoring and recency so uh as human decision makers we are very clearly anchored on more recent outcomes rather than past outcomes and that actually

again if you have a a Time series that changes quite a bit um that's not necessarily a bad thing right that's exactly what what techniques like exponential smoothing are doing is that they're basically anchored on on recent

outcomes but if you have relatively stable time series where the patterns don't change that much then that creates um an anchoring on on recent observations which is can be very

biasing uh the second bias I mentioned over here is representativeness and to some degree I mentioned this earlier um if you ask people to basically forecast a Time

series you give them time series data and you ask them to forecast the next points they act like as if they're simulating the time series they expect that the sequence of forecasts should

look like what they see in the sequence of data right and as we discussed before that's really not the case forecasting is a uh is is ultimately about noise

reduction right um people see patterns in Randomness if you give people random walk data they will inevitably say oh there's a trend here uh where really all

they're picking up is that random walk randomly walked into the same direction for three periods in a row which happens fairly frequently in a random Mo so people see patterns and

Randomness and they also tend to be over precise right if they uh if you ask for judgmental uh prediction intervals they tend to be narrower than the true ped prediction intervals so those are kind

of individual biases um but there's also organizational biases right the the um forecast is um a coordinating tool

across many different functional areas uh that all want to influence each other through the forecast right so um I think most organizations have moved away from

you know asking salespeople for forecasts but uh that was a common practice just kind of 10 to 20 years ago right and then it leads to all kinds of

biases where you know salespeople might either think that oh my my goal is going to my sales targets are going to depend on the forecast so if I lowball the

forecast I'm going to get lower targets that are easier to accomplish or they're going to think well um the forecast is going to drive the production quantities that

operations is going to put in place I want more stuff to sell so I'm going to increase the forecast so that you know um production quantities are higher and

I can sell more stuff right so so those are sort of typical um organizational incentives and um you know uh there are

plenty of of so the forecasting process is ultimately a poli itical process uh which is why you know we we have things like sales and operations planning

processes in place that try to rationalize this um and really uh but ultimately um again I think every foraster needs to be aware um that the

forecast can be influenced by um organizational politics next slide bman you're muted and also if we want to

take some questions uh give it a bit of time sure yeah so I think yeah the so most of us in this call we have been doing probably forecasting and we know

that sometimes it is actually easy to become frustrated because um the forecast we produce doesn't probably match the reality as we we think it should and

um and then um so the forecasting process seem to fail really so in actually one of the last chapter of the book we discuss this specific topic and uh discuss what failure means in the

context of forecasting so here I um present a simple uh time plot so in the y- axis we have the demand for a service and the x-axis we have the month so the

black line is the actual um which seems to be actually quite consistent so there are consistent patterns in the in the historical time series and the forecast

is produced for 48 months so the blue line is the point forecast then 80% 95% prediction intervals and many other um different possible Futures there and

then of course most of us we know something um happened here we had coid 19 um and then uh it would be interesting to see actually what uh the

how the model performs when it comes to this and it is pretty obvious that um after coid um the demand for this service was almost zero the model

actually was doing pretty a pretty good job up to here as we can see but then it continues to do the same thing because it assumes that the the features or

patterns observed in the past you'll be observed in the future as well but that was not the case so the basically this is an example of failure so in the book we discuss different aspect of it uh we

discuss actually we don't have any other alternative to forecasting so we have to make decision and plan based on that so that's why we need actually to do the forecast so that's why the forecast

doesn't really fail but we need to understand probably different aspect of it um if you are dealing with um let's say a variable that is um uh dominated

by noise so well it is likely that you can get something that is accurate if you make decisions only based on the point

forecast ignoring uncertainty again um this might not be very helpful um and well uh the rest of it so um I think as

even said maybe I will stop here and we can take some questions and go if you want to buy the book as well there is a discount that you can use um uh to buy

the physical copy of the book great thanks a lot uh thanks for managing to put so much information in such a short uh

presentation we have actually several questions in the chat actually in fact we I see that several people are discussing that already on their own but it would be nice you know to bring some

of those questions um to the wide audience so the first was uh question from Sven where would you place human judgment with regard to interpretability

and complexity I've seen that Eno has already replied something but it would be interesting to hear each one of you if you have something to say about that

um who wants maybe en know you can vocalize what you've written and then we'll move to the yeah I mean it's it's sort of that depends right I mean um if

if human judgment in an organization is really just like okay everybody's free for all adjust the forecast as you will without providing any reasons then it's

um you know neither interpretable not particularly simple um so but you can you can set up a process for this right

um so I'm a big fan of you know in some sense not having people adjust the forecast but having people just provide reasons of why the forecast might be

wrong right so classic example is hey we have scheduled a promotion and I don't think the forecast reflects the promotion well that's important information I mean first of all that should be a feedback to your data

scientists that says why does the forecast not include a promotion that we've scheduled right and second of all well that's a fairly straightforward adjustment that um you know once you

know that there's a promotion scheduled you can estimate what that means and have an algorithm basically do the adjustment so I'm a big fan of just having people basically point out the

incompleteness in models and um you can set up a process for that that is both in very interpretable and not complex at all and I think really quite

effective bman St do you have anything to to add when it comes to judgmental adjustments uh sorry I I was answering a

different question in the chat and we're talking about just different judgmental adjustments aren't we yep yeah yeah okay so from my from our experience it's it's

often also a case of uh people trusting the system more so what we've seen is when we Implement our solution at a customer then the amount of the number of judgmental adjustment really goes

down over time uh that may reflect perhaps the system is getting better because the data are getting better and the the the entire process is is kind of getting smoother and smoother or what I

like to think is that people learn to trust the system more and more so it's it's also it's really a question is is there somebody who's feeling uh their

their personal identity lies in forecasting then they'll have an issue if a like a machine comes over and and takes over or is there somebody else

who's identity is something else is perhaps getting the supply chain to run smoothly and forecasting is something they had to do on the side or as a as a means to an end people like those will

be happier to let the machine take over and they'll not feel so invested and not feel uh the the compulsion to really change matter so it's really a question of the process that you're putting the

forecast in and about the people that are in there okay thanks right uh well if you don't have anything to

add bman then I'll move to the next question I think sorry just one one thing that maybe it is relevant here is the sort of like decision- making process that follows the forecast so of

course is it's if you have an automatic decision- making process there in place so maybe there is less than actually judge adjusting the forecast itself it

goes maybe to the decision it could be sometimes the case uh but again again in many cases that I came across as well the decision making is manual and uh the

the forecast judgment happens and I think ano and Stefan they cover it both yeah right thanks a lot uh there is a

question related to one of the very hypy topic recent ones uh so what is your take on conformal prediction in intervals I'll I'll phrase the question like

that perhaps I already gave a little answer in there so I personally don't understand it yet I hope we'll understand it at some point in time because we really need to include it in the book because it's all the rage

nowadays uh just Googling for it gives you a commentary by Tim yski on the M5 competition where he essentially says in a throwaway remark that most of the top scorers on the M5 competition used

conformal prediction intervals which he said was pretty much just just uh bootstrapping or resampling in Sample

residuals so that's nice um it's but it's not really completely rocket science it would not be the first uh the first time that something that's not

completely that's not huge Advance would be hyped by by lots of people who think it's it's wonderful my personal question always is I'm I'm always very concerned

about uh not only differences in the mean but also differences in the variant so with forecasts need to be to take account of hros Asis because sometimes

you know that your your quantal should be farther away from the mean and sometimes less far because you have different uh variances in the process of forecasting I'm always a bit concerned

about whether conformal prediction intervals can deal with that apparently there are newer methods and approaches that can deal with that it's just that yeah we need to understand that a little

more a little better but honestly I haven't seen anybody in the M5 competition who trumpeted that they were using conformal prediction and that made the difference

and so not quite sure whether that's because there were no libraries at that point or because uh the kaggle community was not

aware of that so it's it Bears watching okay and bahman do you have anything to add to this no okay uh well we are running out

of time so I'll actually pick a random question from the chat is the final one any suggestions for upior for castability measures I think and this is more of a

question to you because you mentioned forast stability in in the chat so can comment actually I gave an answer to that uh perhaps I can just read my

answer coefficient of variation or anything else is pretty much useless and I personally always say the proof of the putting is in the eating and the proof of the forecast is in the quality uh so

it's very hard to assess forecast ability ahead of time because you don't know what you don't know and if somebody else knows something that you don't know then they might be much they might find

a a Time series much easier to forecast than somebody than you might be so forecast stability is never a a function of the for time series itself it's always a function of the information set

of the knowledge of the forecaster so given the certain amount of knowledge as time series is easier or harder to forecast so and that ties back into understanding your data which we talked about at the very beginning uh if you

can expand your knowledge your information set about the time series and a four time series become suddenly may become much more easily forecastable and that's very hard to say in

general just uh just add to what Stefan said I think well in general uh coefficient of variation or entropy might be used to say what is the ratio of signal to noise or something like

that but uh I agree with Stefan uh so most of the time actually these measures they just look at the time series itself but U there are a lot of other things that affect the time series if you spend

time understand what are actually those driving factors those events and you collect the data for it at that time series that maybe the coefficient of variation or entropy told you it is

difficult to forecast actually it becomes really maybe easy to forecast so I think that's that domain knowledge collecting relevant uh predictors and building in the model

I think is is the key there okay I think a lot of it has to do with understanding the market structure right if you're predicting the behavior

of very few very large customers that place huge orders then you can have a lot of unpredictability in the market right versus if you at an aggregate

level have many small customers then you have a high level of aggregation and and that's much easier in some sense to predict so sort of understanding your Market structure is maybe the the first

way of of um categorizing forecastability of course in general scale right uh do you predict sort of slow moving items where you just have like zeros and ones throughout the year

or do you have something that is that that moves kind of faster which again is is related to Market structure so I think for me understanding the customer base and the level of aggregation that

you want to predict is probably key in terms of understanding forecastability right well uh using my own power I'll

ask you a bonus question but it will be very small one how can you handle pressure to make the forecast the same as the Strategic plan that's the question from

Caroline and I'll use my superpower I already answered that one yeah in the book we we pay very much attention to the difference between a forecast a plan

and a target a forecast is what we expect the future to be like that's our expectation based on our actions of course uh the the target is what we want

to achieve and we tailor our actions to reaching that Target and the plan is what we plan for is the decisions we make to to address the future that will come along and there is a difference

here and of course educating people is easier said than done I see that every day when I try to educate my children and that's again a point where communication comes in that's the short

answer because we're running out of time great I think we can finish uh finish now thanks a lot for the presentation thank you everyone for your

questions the chat was very very Lively uh this time so we will have another um event another webinar next month in

November so so stay tuned and once again thank you bahman and and Stefan and see you around thank you everybody it's been a

wonderful time goodbye thank you byebye