Category: Data Science Methodology

Deep Vector Autoregression for Macroeconomic Data

Data Science master project by Marc Agustí, Patrick Altmeyer, and Ignacio Vidal-Quadras Costa ’21

Photo by Uriel SC on Unsplash

Editor’s note: This post is part of a series showcasing BSE master projects. The project is a required component of all Master’s programs at the Barcelona School of Economics.

Abstract

Vector autoregression (VAR) models are a popular choice for forecasting of macroeconomic time series data. Due to their simplicity and success at modelling the monetary economic indicators VARs have become a standard tool for central bankers to construct economic forecasts. Impulse response functions can be readily retrieved and are used extensively to investigate the monetary transmission mechanism. In light of the recent advancements in computational power and the development of advanced machine learning and deep learning algorithms we propose a simple way to integrate these tools into the VAR framework.

This paper aims to contribute to the time series literature by introducing a ground-breaking methodology which we refer to as DeepVector Autoregression (Deep VAR). By fitting each equation of the VAR system with a deep neural network, the Deep VAR outperforms the VAR in terms of in-sample fit, out-of-sample fit and point forecasting accuracy. In particular, we find that the Deep VAR is able to better capture the structural economic changes during periods of uncertainty and recession.

Conclusions

To assess the modelling performance of Deep VARs compared to linear VARs we investigate a sample of monthly US economic data in the period 1959-2021. In particular, we look at variables typically analysed in the context of the monetary transmission mechanism including output, inflation, interest rates and unemployment.

Our empirical findings show a consistent and significant improvement in modelling performance associated with Deep VARs. Specifically, our proposed Deep VAR produces much lower cumulative loss measures than the VAR over the entire period and for all of the analysed time series. The improvements in modelling performance are particularly striking during subsample periods of economic downturn and uncertainty. This appears to confirm or initial hypothesis that by modelling time series through Deep VARs it is possible to capture complex, non-linear dependencies that seem to characterize periods of structural economic change.

Chart shows that improvement in performance of Deep VAR over VAR model
Credit: the authors

When it comes to the out-of-sample performance, a priori it may seem that the Deep VAR is prone to overfitting, since it is much less parsimonious that the conventional VAR. On the contrary, we find that by using default hyperparameters the Deep VAR clearly dominates the conventional VAR in terms of out-of-sample prediction and forecast errors. An exercise in hyperparameter tuning shows that its out-of-sample performance can be further improved by appropriate regularization through adequate dropout rates and appropriate choices for the width and depth of the neural. Interestingly, we also find that the Deep VAR actually benefits from very high lag order choices at which the conventional VAR is prone to overfitting.

In summary, we provide solid evidence that the introduction of deep learning into the VAR framework can be expected to lead to a significant boost in overall modelling performance. We therefore conclude that time series econometrics as an academic discipline can draw substantial benefits from further work on introducing machine learning and deep learning into its tool kit.

We also point out a number of shortcomings of our paper and proposed Deep VAR framework, which we believe can be alleviated through future research. Firstly, policy-makers are typically concerned with uncertainty quantification, inference and overall model interpretability. Future research on Deep VARs should therefore address the estimation of confidence intervals, impulse response functions as well as variance decompositions typically analysed in the context of VAR models. We point to a number of possible avenues, most notably Monte Carlo dropout and a Bayesian approach to modelling deep neural networks. Secondly, in our initial paper we benchmarked the Deep VAR only against the conventional VAR. In future work we will introduce other non-linear approaches to allow for a fairer comparison.

Code

To facilitate further research on Deep VAR, we also contribute a companion R package deepvars that can be installed from GitHub. We aim to continue working on the package as we develop our research further and want to ultimately move it onto CRAN. For any package related questions feel free to contact Patrick, who authored and maintains the package. The is also a paper specific GitHub repository that uses the deepvars package.

Download the full paper
Get the R package

Connect with the authors

About the BSE Master’s Program in Data Science Methodology

Individual recourse for Black Box Models

Explained intuitively by Patrick Altmeyer (Finance ’18, Data Science ’21) through a tale of cats and dogs

Is artificial intelligence (AI) trustworthy? If, like me, you have recently been gobsmacked by the Netflix documentary Coded Bias, then you were probably quick to answer that question with a definite “no”. The show documents the efforts of a group of researchers headed by Joy Buolamwini, that aims to inform the public about the dangers of AI.

One particular place where AI has already wreaked havoc is automated decision making. While automation is intended to liberate decision making processes of human biases and judgment error, it all too often simply encodes these flaws, which at times leads to systematic discrimination of individuals. In the eyes of Cathy O’Neil, another researcher appearing on Coded Bias, this is even more problematic than discrimation through human decision makers because “You cannot appeal to [algorithms]. They do not listen. Nor do they bend.” What Cathy is referring to here is the fact that individuals who are at the mercy of automated decision making systems usually lack the necessary means to challenge the outcome that the system has determined for them. 

In my recent post on Towards Data Science,  I look at a novel algorithmic solution to this problem. The post is based primarily on a paper by Joshi et al. (2019) in which the authors develop a simple, but ingenious idea: instead of concerning ourselves with interpretability of black-box decision making systems (DMS), how about just providing individuals with actionable recourse to revise undesirable outcomes? Suppose for example that you have been rejected from your dream job, because an automated DMS has decided that you do not meet the shortlisting criteria for the position. Instead of receiving a standard rejection email, would it not be more helpful to be provided with a tailored set of actions you can take in order to be more successful on your next attempt? 

The methodology proposed by Joshi et al. (2019) and termed REVISE is an attempt to put this idea into practice. For my post I chose a more light-hearted topic than job rejections to illustrate the approach. In particular, I demonstrate how REVISE can be used to provide individual recourse to Kitty 🐱, a young cat that identifies as a dog. Based on information about her long tail and short overall height, a linear classifier has decided to label Kitty as a cat along with all the other cats that share similar attributes (Figure below). REVISE sends Kitty on the shortest possible route to being classified as a dog 🐶 . She just needs to grow a few inches and fold up her tail (Figure below).

The following summary should give you some flavour of how the algorithm works:

  1. Initialise x, that is the attributes that will be revised recursively. Kitty’s original attributes seem like a reasonable place to start.
  2. Through gradient descent recursively revise x until g(x*)=🐶. At this point the descent terminates since for these revised attributes the classifier labels Kitty as a dog.
  3. Return x*-x, that is the individual recourse for Kitty.
Animation illustrates how Kitty crosses the decision boundary
The simplified REVISE algorithm in action: how Kitty crosses the decision boundary by changing her attributes. Regularisation with respect to the distance penalty increases from top left to bottom right. Image by author.

This illustrative example is of course a bit silly and should not detract from the fact that the potential real-world use cases of the algorithm are serious and reach many domains. The work by Joshi et al. adds to a growing body of literature that aims to make AI more trustworthy and transparent. This will be decisive in applications of AI to domains like Economics, Finance and Public Policy, where decision makers and individuals rightfully insist on model interpretability and explainability. 

Further reading

The article was featured on TDS’ Editor’s Picks and has been added to their Model Interpretability column. This link takes you straight to the publication. Readers with an appetite for technical details around the implementation of stochastic gradient descent and the REVISE algorithm in R may also want to have a look at the original publication on my personal blog.

Connect with the author

portrait


Following his first Master’s at Barcelona GSE (Finance Program), Patrick Altmeyer worked as an economist for the Bank of England for two years. He is currently finishing up the Master’s in Data Science at Barcelona GSE.

Upon graduation Patrick will remain in academia to pursue a PhD in Trustworthy Artificial Intelligence at Delft University of Technology.

How we used Bayesian models to balance customer experience and courier earnings at Glovo

Javier Mas Adell ’17 (Data Science)

Neon sign depicts Bayes' Theorem

Glovo is a three-sided marketplace composed of couriers, customers, and partners. Balancing the interests of all sides of our platform is at the core of most strategic decisions taken at Glovo. To balance those interests optimally, we need to understand quantitatively the relationship between the main KPIs that represent the interests of each side.

I recently published an article on Glovo’s Engineering blog where I explain how we used Bayesian modeling to help us tackle the modeling problems we were facing due to the inherent heterogeneity and volatility of Glovo’s operations. The example in the article talks about balancing interests on two of the three sides of our marketplace: the customer experience and courier earnings.

The skillset I developed during the Barcelona GSE Master’s in Data Science is what’s enabled me to do work like this that requires knowledge of machine learning and other fields like Bayesian statistics and optimization.

Read the full article on Glovo’s Engineering blog

Connect with the author

portrait

Javier Mas Adell ’17 is Lead Data Scientist at Kannact. He is an alum of the Barcelona GSE Master’s in Data Science.

Stop dropping outliers, or you might miss the next Messi!

Jakob Poerschmann ’21 explains how to teach your regression the distinction between relevant outliers and irrelevant noise

Jakob Poerschmann ’21 (Data Science) has written an article called “Stop Dropping Outliers! 3 Upgrades That Prepare Your Linear Regression For The Real World” that was recently posted on Towards Data Science.

The real world example he uses to set up the piece will resonate with every fan of FC Barcelona (and probably scare them, too):

You are working as a Data Scientist for the FC Barcelona and took on the task of building a model that predicts the value increase of young talent over the next 2, 5, and 10 years. You might want to regress the value over some meaningful metrics such as the assists or goals scored. Some might now apply this standard procedure and drop the most severe outliers from the dataset. While your model might predict decently on average, it will unfortunately never understand what makes a Messi (because you dropped Messi with all the other “outliers”).

The idea of dropping or replacing outliers in regression problems comes from the fact that simple linear regression is comparably prone to extremes in the data. However, this approach would not have helped you much in your role as Barcelona’s Data Scientist. The simple message: Outliers are not always bad!

Dig into the full article to find out how to prepare your linear regression for the real world and avoid a tragedy like this one!

Read “Stop dropping outliers” on Towards Data Science

Connect with the author

portrait

Jakob Poerschmann ’21 is student in the Barcelona GSE Master’s in Data Science.

Data Science team “Non-Juvenile Asymptotics” wins 3rd prize in annual Novartis Datathon

Patrick Altmeyer, Eduard Gimenez, Simon Neumeyer and Jakob Poerschmann ’21 competed against 57 teams from 14 countries.

Screenshot of team members on videoconference
Members of the “Non-Juvenile Asymptotics” Eduard Gimenez, Patrick Altmeyer, Simon Neumeyer and Jakob Poerschmann, all Barcelona GSE Data Science Class of 2021

The Novartis Datathon is a Data Science competition taking place annually, usually in Barcelona. In 2020, the Barcelona GSE team “Non-Juvenile Asymptotics” consisting of Eduard Gimenez, Patrick Altmeyer, Simon Neumeyer and Jakob Poerschmann won third place after a fierce competition against 57 teams from 14 countries all over the globe. While the competition is usually hosted in Barcelona, the Covid-friendly version was fully remote. Nevertheless, the increased diversity of teams clearly made up for the missed out atmosphere.

This year’s challenge: predict the impact of generic drug market entry

The challenge of interest concerned predicting the impact of generic drug market entry. The risk of losing ground against cheaper drug replicates once the patent protection runs out is evident for pharmaceutical companies. The solutions developed helped solving exactly this problem, making drug development much easier to plan and calculate.

While the problem could have been tackled in various different ways, the Barcelona GSE team focused on initially developing a solid modeling framework. This represented a risky extra effort in the beginning. In fact more than half of the competition period passed without any forecast submission by the Barcelona GSE team. However, the initial effort clearly paid off: as soon as the obstacle was overcome, the “Non-Juvenile Asymptotics” were able to benchmark multiple models at rocket speed.

Fierce competition until the very last minute

The competition was a head-to-head race until the last minute. Still in first place until minutes before the final deadline, the predictions of two teams from Hungary and Spain ended up taking the lead by razor sharp margins.

Congratulations to the winners!!!

Group photo of the team outside the entrance of Universitat Pompeu Fabra
The team at Ciutadella Campus (UPF)

Connect with the team

Tracking the Economy Using FOMC Speech Transcripts

Data Science master project by Laura Battaglia and Maria Salunina ’20

Editor’s note: This post is part of a series showcasing BSE master projects. The project is a required component of all Master’s programs at the Barcelona School of Economics.

Abstract

In this study, we propose an approach for the extraction of a low-dimensional signal from a collection of text documents ordered over time. The proposed framework foresees the application of Latent Dirichlet Allocation (LDA) for obtaining a meaningful representation of documents as a mixture over a set of topics. Such representations can then be modeled via a Dynamic Linear Model (DLM) as noisy realisations of a limited number of latent factors that evolve with time. We apply this approach to Federal Open Market Committee (FOMC) speech transcripts for the period of Greenspan presidency. This study serves as exploratory research for the investigation into how unstructured text data can be incorporated into economic modeling. In particular, our findings point at the fact that a meaningful state-of-the-world signal can be extracted from expert’s language, and pave the way for further exploration into the building of macroeconomic forecasting models, and in general into the usage of variation in language for learning about latent economic conditions.

Key findings

In our paper, we develop a sequential approach for the extraction of a low-dimensional signal from a collection of documents ordered over time. We apply this framework to the US Fed’s FOMC speech transcripts for the period 08-1986 to 01-2006. We retrieve estimates for a single latent factor, that seem to track fairly well a specific set of topics connected with risk, uncertainty, and expectations. Finally, we find a remarkable correspondence between this factor and the Economic Policy Uncertainty Indices for United States.

figure
figure
Download the full paper

Connect with the authors

About the BSE Master’s Program in Data Science Methodology

Structure and power dynamics in labour flow and company control networks in the UK

Data Science master project by Áron Pap ’20

Droplets of dew collect on a spider web
Photo by Nathan Dumlao on Unsplash

Editor’s note: This post is part of a series showcasing BSE master projects. The project is a required component of all Master’s programs at the Barcelona School of Economics.

Abstract

In this thesis project I analyse labour flow networks, considering both undirected and directed configurations, and company control networks in the UK. I observe that these networks exhibit characteristics that are typical of empirical networks, such as heavy-tailed degree distribution, strong, naturally emerging communities with geo-industrial clustering and high assortativity. I also document that distinguishing between the type of investors of firms can help to better understand their degree centrality in the company control network and that large institutional entities having significant and exclusive control in a firm seem to be responsible for emerging hubs in this network. I also devise a simple network formation model to study the underlying causal processes in this company control network.

figure

Conclusion and future research

Intriguing empirical patterns and a new stylized fact are documented during the study of the company control network, since there is suggestive evidence that the types and number of investors are strongly associated with how “interconnected” a firm is in the company control network. Based on the empirical data it also seems that the largest institutional investors mainly seek opportunities where they can have significant control without sharing it with other dominant players. Thus the most “interconnected”/central firms in the company control network are the ones who can maintain this power balance in their owner structure. 

The devised network formation model helps to better understand the potential underlying mechanisms for the empirically observed stylized facts about the company control network. I carry out numerical simulations, sensitivity analysis and also calibrate parameters of the model using Bayesian optimization techniques to match the empirical results. However, these results could be “fine-tuned” at different stages further, in order to have a better empirical fit. First, the network formation model could be enhanced to represent more complex agent interactions and decisions. But also, the model calibration method could be extended to include more parameters and a larger valid search space for each of those parameters.

This project could also benefit from improvements to the utilised data. For example more granular data on the geographical regions could help to understand the different parts of London more and to have a more detailed view of economic hubs in the UK. Moreover, the current data source provides a static snapshot of the ownership and control structure of firms. Panel data on this front could enhance the analysis of the company control network, numerous experiments related to temporal dynamics could be carried out, for example link prediction or testing whether investors follow some kind of “preferential attachment” rules when acquiring significant control in firms.

Download the full paper

Connect with the author

Áron Pap, Visiting Student at The Alan Turing Institute

About the BSE Master’s Program in Data Science Methodology

Scalable Inference for Crossed Random Effects Models

Data Science master project by Maximilian Müller ’20

Editor’s note: This post is part of a series showcasing BSE master projects. The project is a required component of all Master’s programs at the Barcelona School of Economics.

Abstract

Crossed random effects models are additive models that relate a response variable (e.g. a rating) to categorical predictors (e.g. customers and products). They can for example be used in the famous Netflix problem, where movie ratings of users should be predicted based on previous ratings. In order to apply statistical learning in this setup it is necessary to efficiently compute the Cholesky factor L of the models precision matrix. In this paper we show that for the case of 2 factors the crucial point to this end is not only the overall sparsity of L, but also the arrangement of non-zero entries with respect to each other. In particular, we express the number of flops required for the calculation of L by the number of 3-cycles in the corresponding graph. We then introduce specific designs of 2-factor crossed random effects models for which we can prove sparsity and density of the Cholesky factor, respectively. We confirm our results by numerical studies with the R-packages Spam and Matrix and find hints that approximations of the Cholesky factor could be an interesting approach for further decrease of the cost of computing L.

Key findings

  • The number of 3-cycles in the fill graph of the model are an appropriate measure of the computational complexity of the Cholesky decomposition.
  • For the introduced Markovian and Problematic Design we can prove sparsity and density of the Cholesky Factor, respectively.
  • For precision matrices created according to a random Erdös-Renyi-scheme the Spam algorithms could not find an ordering that would be significantly fill-reducing. This indicates that it might be hard or even impossible to find a general ordering rule that leads to sparse Cholesky factors.
  • For all observed cases, many of the non-zero entries in the Cholesky factor are either very small or exactly zero. Neglecting these small or zero values could spare computational cost without changing the Cholesky factor ‘too much’. Approximate Cholesky methods should therefore be included in further research.
Fill-in-ratio (a measure of relative density of the Cholesky factor) vs. matrix size I for the random Erdös-Renyi scheme. For all permutation algorithms the fill-in-ratio grows linearly in I indicating that in general it might be hard to find a good, fill-reducing permutation.
Download the paper

Connect with the author

About the BSE Master’s Program in Data Science Methodology

Machine Learning for the Sustainable Management of Main Water Supply Assets

Maryam Rahbaralam ’19 (Data Science)

big data

Maryam Rahbaralam ’19 (Data Science) presented “Machine Learning for the Sustainable Management of Main Water Supply Assets” with Jaume Cardús (Aigües de Barcelona) during the Pioneering Fields and Applications (Strong AI) session at the 2019 Big Data and AI Congress in Barcelona.

Abstract

The developed machine learning model gives the prediction of the probability of failure for each pipe section of the water supply network, allowing an early renewal of those in more detrimental conditions in terms of social, environmental and economic consequences.

Video

Maryam Rahbaralam ’19 is a Data Scientist at the Barcelona Supercomputing Center (BSC). She is an alum of the Barcelona GSE Master’s in Data Science.

LinkedIn | Twitter

Solving data science problems with Record Matching

Presentation by Data Science alum Jordan McIver ’15

presentation

Every organisation needs to be able to properly connect disparate datasets to take full advantage of their data assets. Alchemmy held an event to discuss approaches and technologies to connect datasets and watchouts to consider once they are connected.

Check out my talk here where we look at an approach that best enables data scientists by partnering them with the other staff who actually hold the context of the data:

Video summary

Most businesses have some or all of the following problems: not enough data science resources for the work required; a large community of data-adjacent staff who have most of the context but are not contributing what they know in the right way; data science problems lacking that same context; algorithms that cannot overcome a lack of data quality or availability of training data. Jordan walks through the use of interactive dashboards where users quality assess the data and this feeds back into the data science process which addresses these problems.

Jordan

Jordan McIver ’15 is Head of Data Consulting at Alchemmy in London. He is an alum of the Barcelona GSE Master’s in Data Science.

LinkedIn